Paper Summary - Data Ingestion for the Connected World

Data Ingestion for the Connected World

John Meehan, Cansu Aslantas, Stan Zdonik, Nesime Tatbul, Jiang Du

Businesses have been using “Big Data” applications to perform timely analytics to make real/near-real time decisions. Effectiveness of these analytics and decisions depends on how quickly necessary data can be extracted, transformed, and loaded from operational platform to analytical platform while ensuring correctness. According to the authors, it is challenging for these latency sensitive “Big Data” applications to do this via traditional ETL processes which are cumbersome and very slow. They propose a new architecture for ETL which they call streaming ETL. Streaming ETL can take the advantages of the push-based nature of a stream processing system.

In this paper, authors have proposed streaming ETL requirements. Streaming ETL must ensure the correctness and predictability of its results. At the same time, a streaming ETL system must be able to scale with the number of incoming data sources and process data in as timely as possible. They have divided the requirements into three categories:

• ETL requirements
• Streaming requirements
• Infrastructure requirements

ETL Requirements (Data Collection + Bulk Loading + Heterogeneous Data Types)

In the case of streaming data sources, data must be collected, queued, and routed to the appropriate processing channels. A data collection mechanism should have the ability to transform traditional ETL data sources into streaming ETL sources. Data collection should scale with the number of data sources. A streaming ETL engine must have the ability to bulk load freshly transformed data into the data warehouse. Streaming ETL engine should have data routing capability to load semantically related data into multiple target systems.

Streaming Requirements (Out-of-Order and Missing Tuples + Dataflow Ordering + Exactly-Once Processing)

When number of data sources and/or data volume is huge, there is a possibility that data may get out of time-stamp order and sometimes data can be missing altogether. Waiting for the things to be sorted out can introduce an unacceptable latency. Authors have proposed to use timeout value and predictive techniques (e.g. regression) to overcome these issues. To improve the performance, streaming ETL should break large batches into smaller ones and large operation also needs to be broken into a number of smaller operations. Streaming ETL must use ordering constraints to ensure that these smaller operations on smaller batches still produce the same result as their larger counter parts. Also, any data migration to and from the streaming ETL engine must occur once and only once.

Infrastructure Requirements (Local Storage + ACID Transactions + Scalability + Data Freshness and Latency)

Any ETL or data ingestion pipeline needs to maintain local storage for temporary staging of new batches of data while they are being prepared for loading into the backend data warehouse. Streaming ETL is no different. Having local storage will also help to ensure the correctness of temporal ordering and alignment of the data. Since streaming ETL engine will be processing multiple stream at once, and each dataflow instance may try to make modifications to the same state simultaneously, it is expected that streaming ETL must follow ACID transaction semantics. Streaming ETL must also ensure that scalability of data ingestion and data freshness.

Streaming ETL Architecture

Authors propose a new architecture based on the above requirements.This new architecture has four primary components:

Data collection: This component has a collection of data collectors. These data collectors primarily serve as messaging queues. Data collectors consume data from different sources, create logical batches of data and push them to the streaming ETL engine.

Streaming ETL: This component contains a range of ETL tools, including data cleaning and transformation operators. Dataflow graph can be created using these operators to massaged the incoming batches of data into normalised data. Once the data has been fully cleaned and transformed, it can be either pushed into data warehouse or pulled by data warehouse.

OLAP backend: This component consists of a query processor and one or several OLAP engines. Each OLAP engine contains its own data warehouse, as well as a delta data warehouse. Both data warehouses have same schema. Streaming ETL engine writes all updates to the delta data warehouse, and OLAP engine periodically merges these updates into the full data warehouse.

Data migrator: Data migrator ensures that no batch of data get lost when it moves from streaming ETL to OLAP backend components. This should also fully support ACID transactions.

Authors have built a proof-of-concept implementation based on this new architecture using Apache Kafka, S-Store, Intel’s BigDAWG polystore, and Postgres.

In this paper, authors have also tried to answer another important question regarding the frequency of the data migration to the data warehouse by a streaming ETL system. There are two methods: push (ingestion engine periodically pushes the data to the warehouse) and pull (warehouse pulls the data from the ingestion engine when it is needed). Authors have run an experiment to test the pros and cons of each method and according to them pulling new data with each query is the best option if the data staleness is the priority. They also suggested that it is better to go for smaller, more frequent migrations in both push and pull scenarios.

Conclusion

Authors think that streaming ETL can be extended to create all-in-one ingestion and analytics engine specifically for time-series data which they call Metronome (time-series ETL). This paper focuses on the functional requirements of streaming ETL. Authors also build a proof-of-concept implementation based on these requirements.

Visualising Software Architecture Effectively in Service Description

Somedays back one of my team members told me about Simon Brown's C4 model. Since then I have been following this to document the software architecture. This presentation is about the diagrams that I draw (or I like to see) in service description based on C4 model.

Visualizing Software Architecture Effectively in Service Description from Sanjoy Kumar Roy

Paper Summary - Prioritizing Attention in Fast Data: Principles and Promise

Prioritizing Attention in Fast Data: Principles and Promise
Peter Bailis, Edward Gan, Kexin Rong, Sahaana Suri
Stanford InfoLab

Processing and interpreting huge volume data that is in motion (fast data) to get timely answer is challenging and sometimes infeasible due to the scarce of resources (both human and computational). Human attention is limited. According to the authors, a new generation analytic system is needed to bridge the gap between limited human attention and growing volume of data. This new type of analytic system will prioritise attention in fast data. In this paper, authors have proposed three design principles that can be used to design and develop such fast data analytic system:

Principle 1: Prioritise Output – The design must deliver more information using less output.

Fast data analytic system should produce fewer and good quality output. If a system produces lot of raw (output) data, then it becomes difficult for a human to give attention. For example, if the end result is to find out which device is producing more problematic records, then it would be ideal if the system can simply return the device id with the count of records rather than producing every raw problematic record. According to the authors – “A few general results are better than many specific results”.

Principle 2: Prioritise Iteration – The design should allow iterative feedback-driven development.

Modern analytics workflows consist of many steps – including feature engineering, model selection, parameter tuning, and performance engineering. It is difficult to get the final model at first attempt. This means that analytics system should empower the end users by giving them necessary tools so that they can improve these steps iteratively based on the feedback. Today this is very labour intensive and time-consuming task. Fast data analytics system should lower this barrier. Fast data system should be designed for modularity and incremental extensibility.

Principle 3: Prioritise Computation – The design must prioritise computation on inputs that most affect its output.

One of the key property of fast data is – not all inputs contribute equally to the output. Therefore, it is waste of valuable computational resource if the system gives equal importance to all inputs. But how will fast data system select these inputs that contribute most to the output? According to authors – “fast data systems should start from the output and work backwards to the input, doing as little work as needed on each piece of data, prioritizing computation over data that matters most”.

MacroBase

Authors have built a new fast data analysis engine called MacroBase based on the principles outlined above. At present MacroBase’s core dataflow pipelines contain a sequence of data ingestion, feature extraction, classification, and explanation operators. These operators perform tasks including feature extraction, supervised and unsupervised classification, explanation and summarisation. MacroBase can process data as it arrives. It can also process data in batch mode.

MacroBase System Architecture

Users can engage at three interface levels with MacroBase:

• Basic: Web based graphical user interface. This one is an easy interface.
• Intermediate: Custom pipelines configuring using Java.
• Advanced: Custom dataflow operators using Java/C++.

These interfaces will enable users of varying skill levels to quickly obtain the initial results and further improve result quality by iteratively refining their analyses. Users can highlight the key performance metrics (like, power drain, latency) and metadata attributes (like, hostname, device id). MacroBase reports explanations of the abnormal behaviour. For example, MacroBase may report that queries running on host 5 are 10 times more likely to experience high latency than the rest of the cluster. MacroBase is currently doing mostly anomaly or outlier detection, it is not doing any deep machine learning training.

Conclusion

Today we collect large volume of data in analytical platform. Some of these data are never read. Sometimes we may go back and analysis these data to find the root cause of the problem after it happened. Moreover, tools that we use to do these kinds of analysis are not easily accessible and process is time consuming. I think, these design principles provide good guidance which can be used to design and build a new generation analytics engine which can process huge volume of data and produce good quality output in timely manner.

More on this can be found:

• MacroBase: Prioritizing Attention in Fast Data
• https://github.com/stanford-futuredata/macrobase

Hypermedia and how to document it effectively

Hypermedia API and how to document it effectively from Sanjoy Kumar Roy

An introduction to OAuth 2

An introduction to OAuth 2 from Sanjoy Kumar Roy

Transaction

Transaction from Sanjoy Kumar Roy

Microservice Architecture Design Principles

Raft Consensus Algorithm

Imagine we have a single node database server that stores a single value. We also have a client that can send a value to the database server.

Coming to agreement (or consensus) on that value is easy with one node. But how do we come to consensus if we have more than one node?

Here we need to use distributed consensus. Distributed consensus (i.e. protocols) allows nodes in an unreliable distributed system to agree on an ordering of events. Raft is a protocol for implementing distributed consensus.

Distributed consensus is typically framed in the context of a replicated state machine, drawing a clear distinction between the state machine (the fault tolerant application), the replicated log and the consensus module.

Replicated State Machine

Replicated state machines are typically implemented using a replicated log. Each server stores a log containing a series of commands, which its state machine executes in order. Each log contains the same commands in the same order, so each state ma- chine processes the same sequence of commands. Since the state machines are deterministic, each computes the same state and the same sequence of outputs.

Consensus algorithm is responsible for keeping the replicated log consistent. The consensus module on a server receives commands from clients and adds them to its log. It communicates with the consensus modules on other servers to ensure that every log eventually contains the same requests in the same order, even if some servers fail. Once commands are properly replicated, each server’s state machine processes them in log order, and the outputs are returned to clients. As a result, the servers appear to form a single, highly reliable state machine.

Replicated state machines are used to solve variety of fault tolerant problems in distributed system.

Properties of consensus algorithm

1. They never return an incorrect result under all non-Byzantine conditions including network delays, partition, packet loss, duplication, reordering
2. They are fully functional as long as any majority of servers are operational and can communicate with each other and with the clients. For example, a cluster of five servers can tolerate the failure of any two servers
3. They do not depend on the timing to ensure the consistency of the logs
4. In the common case, a command can complete as soon as a majority of the servers has responded to a single round of remote procedure calls. A minority of slow servers do not have impact on the overall system performance.

Now we will see how Raft works. Before that we need to make ourself familiar with some Raft concepts.

Raft is a consensus algorithm for managing replicated log. Raft uses strong leadership. At first Raft selects a leader with the complete responsibility for managing the replicated log. The leader accepts the log entries from the clients, replicates the log entries to the other servers and tells them when it is safe to apply these log entries to their state machines. When a leader fails and becomes disconnected from other servers, a new leader gets elected. Clients are external to the system and must contact the leader directly to communicate with the system.

Raft cluster

Typically Raft cluster is set up using five nodes, so that the system can tolerate two failures.

Server States

According to Raft protocol, a node can be one of three states:

• Follower: A follower is a passive node. It does not issue any request on its own but simply responds to the requests from the leader and the candidates
• Candidate: A candidate is an active node which is attempting to become a Leader. It initiates a request for votes from other nodes. A candidate that receives votes from a majority of the full cluster becomes the new leader
• Leader: Leader node is an active node which is currently leading the cluster. This node handles requests from clients. If a client contacts a follower, it redirects the client to the leader

How does Raft detect obsolete information such as stale leader? Raft detects this using a concept called term.

Term

Term is an arbitrary length of time. Terms are numbered with consecutive integers. Terms act as logical clock in Raft. Each term begins with an election in which one or more candidates attempt to become leader. If a candidate wins the election, then it serves as leader for the rest of the term. There may be a situation in which a term ends with no leader, in this case a new term begins with a new election. Raft ensures that there is at most one leader in a given term.

Each server stores its perspective of the term in persistent storage, which increases monotonically over time. A server’s term is only updated when it starts (or restarts) an election, or when it learns from another server that its term is out of date. All messages include the source server’s term. The receiving server checks it, with two possible outcomes: if the receiver’s term is larger, a negative response is sent, while if the receiver’s term is smaller than or equal to the source’s, its term is updated before parsing the message.

Types of messages

Raft servers communicate with each other using remote procedure calls (RPCs). There are three types of message used in Raft:

• RequestVote: this message is used by the candidates during the election.
• AppendEntries: this message is initiated by the leader to replicate the log entries and to provide a form of heartbeat to the followers.
• InstallSnapshot: this message is used by the leader to send a snapshot of it’s log to the followers that are too far behind.

Leader Election

In Raft, there are two timeout settings which control elections. First is the election timeout. The election timeout is the amount of time a follower waits until becoming a candidate. The election timeout is randomized to be between 150ms and 300ms. After the election timeout the follower becomes a candidate and starts a new election term and votes for itself and sends out RequestVote messages to other servers.

If the receiving server hasn't vote yet in this term then it votes for the candidate and the server resets it's election timeout. Once a candidate has a majority of votes it becomes leader. The leader begins sending out AppendEntries messages to its followers. These messages are sent in intervals specified by the heartbeat timeout. Followers then respond to each AppendEntries message. This election term will continue until a follower stops receiving heartbeats and become a candidate. Requiring a majority of votes guarantees that only one leader can be elected per term.

This process is called Leader Election. All the changes to the system will now go through the leader.

There may be a situation when a candidate neither wins nor loses the election. For example, two followers become candidates at the same time and votes could be split so that no candidate obtains a majority. When this happens, each candidate will time out and start a new election by incrementing its term and initiating another round of RequestVote messages. Raft uses randomized election timeouts to ensure that split votes are rare and that they are resolved quickly.

Log Replication

Once a leader is elected, the leader needs to replicate all changes to the system to all servers. This is done by using the same AppendEntries message that are used for heartbeats.

First a client sends a change to the leader. The change is appended to the leader's log. This log entry is currently uncommitted so it will not update the leader server value. Leader then sends the change to the followers on the next heartbeat.

The change gets replicated in the followers' logs.

An entry is committed on the leader server once a majority of followers acknowledge it.

After receiving the acknowledgement from the followers, leader commits the entry.

The leader then notifies the followers that the entry is committed.

The cluster has now come to consensus about the system state and leader sends the response to the client.

This process is called Log Replication.

Now, consider the case that some messages have been lost or servers have failed and recovered, leaving some logs incomplete. It is the responsibility of the leader to fix this by replicating its log to all other servers. When a follower receives an AppendEntries message, it contains the log index and term associated with the previous entry. If this does not match the last entry in the log, the follower sends an unsuccessful response to the leader. The leader is now aware that the follower's log is inconsistent and needs to be updated. The leader decrements the previous log index and term associated with that server. The leader keeps dispatching the AppendEntries message, adding entries to the log until the follower server replies with success and is therefore up to date.

Each server keeps its log in persistent storage, including a history of all commands and their associated terms. Each server also has a commit index, which represents the most recent command to be applied to the replicated state machine. When the commit index is updated, the server passes all commands between the new and old commit index to the local application state machine.

Safety

Raft uses State Machine Safety properties:

• Election Safety: at most one leader can be elected in a given term.
• Leader Append-Only: a leader never overwrites or deletes entries in its log; it only appends new entries.
• Log Matching: if two logs contain an entry with the same index and term, then the logs are identical in all entries up through the given index.
• Leader Completeness: if a log entry is committed in a given term, then that entry will be present in the logs of the leaders for all higher-numbered terms.
• State Machine Safety: if a server has applied a log entry at a given index to its state machine, no other server will ever apply a different log entry for the same index.

Conclusion

The authors of Raft focus on the understandability. Raft is designed to be easy to understand. According to the authors:

"...our most important goal—and most difficult challenge—was understandability. It must be possible for a large audience to understand the algorithm comfortably. In addition, it must be possible to develop intuitions about the algorithm, so that system builders can make the extensions that are inevitable in real-world implementations."

I like the effort Raft's authors put to make the algorithm understandable. They have given many talks and created course materials. All these you can find here https://raft.github.io/

References

• Raft paper for detail explanation
• Raft Refloated: Do We Have Consensus? by Heidi Howard, Malte Schwarzkopf, Anil Madhavapeddy, and Jon Crowcroft
• ARC: Analysis of Raft Consensus by Heidi Howard

My talk on agile architecture in Agile Manchester 2015

I have given a talk on agile architecture in Agile Manchester 2015. Here are the slides:

Modularity

One of the challenging part of any software design is to manage complexity. By managing complexity effectively we can respond to change quickly, improve time to market, reduce cost of change and also improve stability of the system. Modularity is useful for managing quality and complexity in software systems.

Modularity is the concept of breaking down a complex problem into smaller, simpler and more manageable problems.

In software design, modularity refers to the extent to which a software application may be divided into smaller modules. The goals of structuring an application or system in modules is to be able to develop, test and deploy them separately. Modularity also enforces separation of concerns by vertically or horizontally partitioning a system. It keeps a clear separation between business functionality and data/information.

One way to achieve modularity is functional decomposition which basically means that each module or sub-system has a clear domain responsibility and is part of a larger eco-system. Each sub-system may consist of one or more independent modules.

Benefits of functional decomposition:

• Increase quality and reduce complexity
• Parallel development and roll-outs
• Horizontal load balancing of functionality
• Reduced dependencies between different functional areas
• Functional areas can scale separately and on demand
• Asynchronous workflows that increase availability and evenly balance out peak loads
• Smaller modules most of the time brings smaller data sets. This helps to reduce database size, database server workloads and simplified ORM mappings.

Also it is really difficult to become agile or apply agile methodologies without a modular system. A modular system helps you to replace, upgrade or throw away certain part of the system without affecting much the rest of the system.

Eventual Consistency

Introduction

Eventual consistency is a consistency model used in distributed computing to achieve high availability that informally guarantees that, if no new updates are made to a given data item, eventually all accesses to that item will return the last updated value.^[1]

To ensure high availability and scalability, distributed system keeps copies of its data across multiple machines (sitting in different data centers). When a change happened to a data item on one machine, that change has to be propagated to the other replicas. The change propagation will not be happened instantly since there is a network delay. This interval of time allows window of inconsistency during which some of the copies will have the most recent change, but others won't. In other words, the copies will be mutually inconsistent. However, the change will eventually be propagated to all the copies. Hence it is called eventual consistency.

When we talk about eventual consistency, we also need to mention CAP Theorem.

CAP Theorem

CAP Theorem was presented by Eric Brewer in a keynote address to PODC (Principles of Distributed Computing) in 2001. CAP Theorem identifies three important properties of distributed system: Consistency, Availability and Partition Tolerance. Out of these three properties, only two can be achieved at a given time.

Since it is impossible simultaneously to achieve always-on experience (availability) and reading the latest written version of data from a distributed database (consistency) in the presence of partial failure (partitions), distributed system architects sacrifice "strong" consistency to ensure availability and partition tolerance. In other way it can be said that they use weaker models and eventual consistency is the most notable one.

Examples

• DNS
• Asynchronous master/slave replication on an RDBMS
• Caching in front of relational databases
• NoSQL databases

Variations of Eventual Consistency

• Causal consistency If process A has communicated to process B that it has updated a data item, a subsequent access by process B will return the updated value, and a write is guaranteed to supersede the earlier write. Access by process C that has no causal relationship to process A is subject to the normal eventual consistency rules. Eventual Consistency does not say anything about the ordering of operations where as causal consistency ensures that operations appear in the order the user intuitively expects. It enforces a partial order over operations.
• Read-your-writes consistency This is an important model where process A, after it has updated a data item, always accesses the updated value and will never see an older value. This is a special case of the causal consistency model.
• Session consistency This is a practical version of the previous model, where a process accesses the storage system in the context of a session. As long as the session exists, the system guarantees read-your-writes consistency. If the session terminates because of a certain failure scenario, a new session needs to be created and the guarantees do not overlap the sessions.
• Monotonic read consistency If a process has seen a particular value for the object, any subsequent accesses will never return any previous values.
• Monotonic write consistency In this case the system guarantees to serialize the writes by the same process. Systems that do not guarantee this level of consistency are notoriously hard to program.

A big advantage of Eventual Consistency is that it is fairly straightforward to implement. To ensure convergence, replicas must exchange information with one another about which writes they have seen. This information exchange process is often called anti-entropy. There are different ways to achieve this. One simple solution is to use an asynchronous all-to-all broadcast. When a replica receives a write to a data item, it immediately responds to the user, then, in the background, sends the write to all other replicas, which in turn update their locally stored data items. In the event of concurrent writes to a given data item, replicas deterministically choose a "winning" value, often using a simple rule such as "last writer wins" (using a clock value embedded in each write).

Even though Eventual Consistency does not make any safety guarantee, eventual consistent data store are widely deployed. Because it is "good enough", given its latency and availability benefits.

References

• http://en.wikipedia.org/wiki/Eventual_consistency
• http://www.allthingsdistributed.com/2008/12/eventually_consistent.html
• Don't Settle for Eventual Consistency- Wyatt Lloyd, Michael J. Freedman, Michael Kaminsky, David G. Andersen
• Eventual Consistency Today: Limitations, Extensions, and Beyond - Peter Bailis and Ali Ghodsi

TierCompilation

TierCompilation is a mix of client (C1) and server (C2) compilation. With tiered compilation, code is first compiled by the client compiler. When it becomes hot, it is recompiled by the server compiler.

The goal of the TierCompilation is to get best of both client (C1) and server (C2) compilers. Client compiler begins compiling sooner than the server compiler does. So client compiler is faster than the server in code execution. But client compiler provides less optimization and code quality is not as good as server generated one.

Though server compiler is slow, it provides better quality code. Server compiler waits to gain the knowledge about the code and uses the knowledge to optimize the code. It inlines much more aggressively. Code produced by the server compiler is faster than that produced by client compiler. TierCompilation takes the advantages of both client (fast startup) and server compiler (peak performance).

In Java 7, -XX:+TieredCompilation flag needs to be used to enable TierCompilation. Make sure you specify the server compiler with the -server flag or by ensuring it is the default for the particular Java installation being used. In Java 8, TierCompilation is enabled by default.

An introduction to G1 (Garbage First) Collector

Microservice architecture

This year QCon 2014 London many speakers mention microservice architecture style. They talk about the problems they face with monolithic system and how they solve the problems using this type of architecture style. Martin Fowler together with James Lewis has written a multiple part article about the characteristics of microservice. Monolithic system has several disadvantages:

• It is difficult to change.
• It takes time to add new feature.
• Difficult to test.
• Deployment is a pain.
• and many more

But it happens to every developer's life. Most of us are still going through this. Business people come to us to have something that may generate money for the business and we start with something small. We keep on adding new features to this system and over the time it becomes a monolithic system with all the characteristics mentioned above. There is another scenario - we spend several months to build something and then realize that this is not what business wants, wasting of valuable time and money. Microservice architecture can help us to overcome these situations.

So what is microservice architecture?

Microservice Architecture is a concept that aims to decouple a solution by decomposing functionality into discrete services.

So rather than putting all functionalities in one application, you go for functional decomposition. Each of the function or capability of the application becomes a service. Each of the service becomes easier to understand, develop, test and deploy. These services can scale and evolve at their own pace.

When developing application using microservice architecture style, we need to think about business capabilities of the application.

Each capability represents a service and has its own bounded context.

An example: Reward Management System

Say ABC company wants to develop this system that gives rewards to their valuable customers when they opt into a offer after seeing it in a campaign and perform some activities written in that offer. Once they complete these activities, the system gives them rewards to make them happy.

Now after performing some brain storming sessions, developers of that company come to a conclusion that the Reward Management System needs to have four capabilities:

• Management of campaigns
• Management of offers
• Tracking of customers' activities
• Give reward

They can develop one application with all capabilities in one place. But since they want to give a try microservice architecture style, so they decide to develop individual service for each of the capabilities. So

• Management of campaigns becomes Campaign Service
• Management of offers becomes Offer Service
• Tracking of customers' activities becomes Tracking Service
• Give reward becomes Reward service

Bounded Context

A Bounded Context is an explicit boundary within which a domain model exists. Inside the boundary all terms and phrases of the Ubiquitous Language have specific meaning, and the model reflects the Language with exactness.

In Reward Management System each of the capabilities has its own bounded context. Campaign Service only deals with campaigns: creating, updating, viewing, deleting, associating offer with a campaign and approving, publishing campaigns. Offer Service manages offers: again creating, updating, viewing, deleting offer. Tracking service tracks which offer a customer has chosen to opt in and how far she or he has completed her or his activities to get the reward. Reward service gives reward once customers complete their activities. All these services talk to each other to reach a common goal which is giving rewards to the customers.

Benefits

This type of architecture style has many benefits:

Reduce complexities to understand

In microservice architecture: One business capability = One Service. So services are not suppose to be very big. Each of these service should not big enough to overcome the thinking process. You need to understand what it does. If any service goes beyond your thinking process, then it may be doing more than one thing. So it may be the right time to break it further. This reduces the complexities to understand a lot and eventually help in thinking process. In Reward Management System example, development team is not thinking the full system, most of the time they are thinking about individual service in isolation.

Easy to change

Since each service is small and focus on one capability of the system, you can change it quite easily. Even after development, if you find that it is not meeting business needs (based on different business relevant metrics), you can throw it and develop it again.

Cross functional teams

A cross-functional team is a group of people with different functional expertise working toward a common goal. Cross functional team is a self-directed team. Assigning a task to a team composed of multi-disciplinary individuals increases the level of creativity and out of the box thinking. Each member offers an alternative perspective to the problem and potential solution to the task. [wiki]

Microservice architecture opens the door for cross functional teams. These teams come with full range of skills for the development: user-experience, back-end development, testing, database etc. They design, build, test and deployed it. This team takes the full responsibility for the software in production.

Choice of technology stack

Same technology stack may not be suitable to solve all types of problem. In microservice architecture different team can select different technology stack for different services based on the problem and different other requirements. For example: one team can go typical java stack (Spring, Hibernate, RDBMS), other team can go for Node.js and NoSQL. There are also many other JVM based languages. Point is, in monolithic architecture, we often stuck with one set of technology whereas microservice architecture gives us many options to choose from. Polyglot programming and polyglot persistence are quite common and easy to do in microservice architecture.

Testing

Since services are small and doing one task, testing becomes easy in microservice architecture. We can add many automated tests that can be run using continuous delivery pipeline.

Real time Monitoring and Metrics

Real time Monitoring and Metrics are an important part of microservice architecture. We need to check both architectural elements (how many requests per second is the database getting) and business relevant metrics (such as how many customers opt into a offer per minute are received). Dashboards can be developed that show up/down status and a variety of operational and business relevant metrics of different services.

Conclusion

There are many other benefits of this architecture style that are not mentioned here. I personally feel that developers can make their professional life easier and less painful by using microservice architecture style. After all, simple solution matters.

Major Java 8 Features

Recently I have given a presentation on major Java 8 features. Here is the presentation

Architectural Views : Context view

When describing the architecture of a software system it is useful to show how the system fits in the existing environment (people, systems and external entities with which it interacts). Context view helps us to do this.

Context view of a system defines the relationships, interactions, dependencies between the system and its environment.

The purpose of the context view is to share the big picture to all stackholders. It answers the questions like:

• What does the system actually do from a functional point of view?
• Who and what other systems are using it?
• How is it related to auxiliary systems or services?

Context diagram is the key model within a context view. It is easy to draw. Just place the system in its environment by relating it to the different actors (users and auxiliary systems) that it interacts with. A context diagram contains the below elements:

• System: the system that is going to be designed. Hide it's internal structure, treat it as a "black box"
• External Entities: these are the auxiliary systems, services, people and groups that the system interacts with
• Connections: theses are interfaces, protocols and connectors that link the external entities and the system being designed.

Here is a sample context diagram:

Characteristics of a productive development team

In Agile Fluency article, Diana Larsen and James Shore have mentioned that Agile teams develop through four distinct stages of fluency. They have defined one star, two star, three star and four star team based on the team Agile fluency. It is an interesting article. If you take all the attributes of one, two and three star team (I will not include four star as it is bit difficult to find a four star team) then you will see that they are the characteristics of a productive development team.

A good and productive development team needs to follow some kind of Agile methods. It may be Scrum or Kanban. Ideally they have a product backlog. They use retrospectives to find out what has gone well and also what hasn't gone well in last iteration. They should keep on doing what has gone well and at same time try to avoid what hasn't gone well. They should write good user stories. A good user story solves many misunderstandings since it involves you to interact with different stackholders of your organization.

Practicing Scrum or Kanban without test-driven development (TDD) is rubbish. Martin Fowler calls it FlaccidScrum. One of the characteristics of a productive team is that they write self testing code. It helps them to reduce their technical debt. Now many teams are also practicing behavior driven development (BDD) along with TDD, which is really good. Main point is, a good team should focus on values and qualities. It is about collective ownership or shared responsibilities.

A productive team simply just don't stop here. They automate their build process. They follow practices that are required to do continuous integration effectively.

Another interesting attribute of a productive team is frequent release. They release as frequently as possible. It helps them to get rapid feedback on new features they have added. These feedbacks are important to build the right product.

Media Types

Contracts define how different parts of a distributed system should interact and media types play an important part in contracts.

 A media type is a combination of formats, processing model, and hypermedia controls.

We can use many standardized media type specifications or create new media types to fit our domain.

Standardized media types (e.g., XHTML or Atom) are well-defined and widely understood. Since many systems support these standardized media types, interoperability between them can be easily achieved by using them.

Custom media types help us to add application specific semantics on the top of generic media types handlers. Jim Webber mentioned a nice example in his blog about custom media type:

"For example, if you get an Atom representation, then you automatically understand (globally) how to interpret the atom:link links within; with a custom hypermedia type (... application/restbucks+xml) you automatically understand (within the Restbucks context) how to interpret links; but for application/xml you have no idea how to extract hypermedia controls unless you have some prior knowledge of the schema."

Reference: REST in Practice - Jim Webber, Savas Parastatidis and Ian Robinson

Characteristics of different levels in Richardson Maturity Model

Recently I am involved in building a Campaign Management System based on Richardson's Level 3 service definition. It's a new thing for my team and we are enjoying it. Leonard Richardson proposed a classification of RESTful web services in his talk. He mentioned four levels in his classification. Richardson evaluated service maturity based on three core technologies: URI, HTTP and Hypermedia. Each layer builds on the concepts and technologies of layers below.

So what are characteristics of these levels?

Level 0 Services

• HTTP is used as a transport system to tunnel requests and responses.
• A single URI.
• Use a single HTTP method (typically POST), ignore the rest of the HTTP verbs.
• Examples: SOAP, XML-RPC and POX (Plain Old XML).

Level 1 Services

• Introduce Resource concept.
• Employ many URIs and each URI acts as an entry point to a specific resource.
• Still a single HTTP verb is used.

Level 2 Services

• At this level services host many URI-addressable resources and also support several of the HTTP verbs on each exposed resources.
• The use of GET for requesting resource is important. HTTP defines GET as safe and idempotent operation. This property of GET help us to optimize the services. When a consumer of a resource uses GET, we know he does not want to modify it. We can use caching to store responses closer to our consumer. Subsequent requests will be served from the caches and that helps in improving the overall quality of the service.
• Another important characteristic is the use of status codes. Services use different status codes to respond. When a resource is created, services respond with a 201 Created whereas 409 Conflict is used to tell that something has gone wrong.

Level 3 Services

• Support HATEOAS (Hypermedia As The Engine Of Application State)
• Now the representations contain URI links. These links may point to other resources (may be interesting to the consumers) or they may represent a transition to a possible future state of the current resource. One important thing to notice here - service tells the consumer what to do next through these links.
• Here the consumer submits an initial request to the entry point of the service. The service handles the request and responds with a resource representation populated with links. The consumer chooses one of these links to transition to the next step in the interaction. Over the course of several such interactions, the consumer progresses toward its goal. In this way the distributed application's state gets changed.
• Consumers in a hypermedia system cause state transitions by visiting and manipulating resource state.

References:

• REST in Practice - Jim Webber, Savas Parastatidis, Ian Robinson
• Martin Fowler's article on Richardson Maturity Model

Self Encapsulation

What is Self Encapsulation?

Martin Fowler mentioned this in his bliki :

"Self Encapsulation is designing your classes so that all access to data, even from within the same class, goes through accessor methods."

This is also called Self Delegation. Take this simple Email example:

public final class Email implements Serializable {
  
  private static final long serialVersionUID = 1L;

  private String emailAddress;

  public Email(String anEmailAddress) {
     super();
     this.setEmailAddress(anEmailAddress);
  }

  public Email(Email anEmail) {
     this(anEmail.getEmailAddress());
  }
  
  public String getEmailAddress() {
     return this.emailAddress;
  }

  private void setEmailAddress(String anEmailAddress) {
     if(anEmailAddress == null) {
        throw new IllegalArgumentException
            ("Email address must not be null.");
     }
     if(anEmailAddress.length() == 0) {
        throw new IllegalArgumentException
            ("Email address is required.");
     }
     if(!java.util.regex.Pattern.matches(
        "\\w+([-+.']\\w+)*@\\w+([-.]\\w+)*\\.\\w+([-.]\\w+)*",
        anEmailAddress)){
        throw new IllegalArgumentException
            ("Email address format is invalid.");
                 
     this.emailAddress = anEmailAddress;
  }
}

In above example, constructor is delegating instance variable, emailAddress assignment to its own internal property setter. Here the setter method is not only setting the email address, but also performing an important assertion. It is providing a guard against invalid data. The self-encapsulation enables the setter method to determine the appropriate contractual condition for setting the email address. This is the advantage of using Self Encapsulation.

References
1. http://martinfowler.com/bliki/SelfEncapsulation.html
2. Implementing Domain-Driven Design - Vaughn Vernon