Distributed transactions: Designing eventual consistent services without Queues
Major cloud providers are pushing for event-driven architecture, leading to a superficial assumption that it’s the ultimate architectural pattern for all applications. However, this push primarily aims to boost the adoption of serverless architecture, which is predominantly event-driven. You might ask, why are they pushing for serverless architecture ? Because Serverless models is appealing to large enterprises, offer the potential for substantial cost savings. The encouragement towards event-driven architecture by cloud providers is thus more about promoting serverless solutions rather than endorsing it as a one-size-fits-all approach.
Adopting event-driven architecture has its advantages, particularly in harnessing the full potential of serverless architecture, which can lead to significant reductions in cloud costs. However, this approach often necessitates deep integration with a specific cloud provider’s services (for example AWS step functions, event bridge, Amazon S3, DynamoDB, Aurora Serverless and so on). This deep integration, while beneficial for leveraging serverless capabilities, can substantially complicate the process of switching cloud providers. Additionally, reliance on these specialized, vendor-specific services may result in vendor lock-in, restricting flexibility and future choices
Before adopting any technology trend, it’s crucial to thoughtfully consider alternatives. This means evaluating why we might not choose synchronous communication or an orchestrated approach over the popular event-driven model. Such reflection ensures that the chosen solution aligns well with specific project requirements and broader organizational goals.
In this blog post I want to propose a non event driven approach to build resilient micro-service architecture.
The approach
In this approach we are going to use synchronous communication, Orchestrated-coordination and eventual consistency. This pattern has a bit of coupling because of synchronous communication and Orchestration. However, worst driver of coupling complexity — transactionality — disappears in this pattern in favor of eventual consistency. The orchestrator must still manage complex workflows, but without the structure of doing so within a transaction.
Below are some benefits of this approach.
Low Complexity
Most beautiful part of this patterns is that complexity is quite low; it includes the most convenient options (orchestrated, synchronicity) with the loosest restriction (eventual consistency).
Automatic load shedding
Automatic load shedding: Queues are durable, meaning that they don’t drop messages even when service is under highload. This can lead to a cascading failures. But synchronous communication, you get load-shedding for free, if service is down or under high load it stop accepting messages.
Easy snapshot Building
Orchestration provides one single place to get the snapshot of the entity. In Choreography, one needs to go to reach out to each service and gather the info about the state. Snapshot building leads to chatty services in choreographed architecture.
Easy troubleshooting
It is very easy to track down where things went wrong, or debugging the whole workflow because of having one single place.
Easy to support various error scenarios
It is very easy to handle all kind of error scenarios in Orchestrated coordination. Choreographed solution are not idea if there are many error scenarios or where micro-services require back and forth communication. Orchestration are very well suited for this.
Orchestration: The right way
When orchestration is employed, it’s common for people to default to the mindset they developed in monolithic architectures, where everything is co-located in a single database, often leading to designs that rely on distributed atomic transactions and rollbacks. The classic example of this is the two-phase commit approach, which essentially superimposes monolithic thinking onto a microservices architecture.
Embracing eventual consistency
However, a more effective strategy in orchestration is to embrace eventual consistency. This approach liberates us from many of the limitations inherent in orchestration, such as the constraints of atomic distributed transactions. By adopting eventual consistency, we can explore more flexible and resilient design patterns that are better suited to the decentralized nature of microservices.
Avoiding leaky abstraction
A further challenge in orchestration arises with failure handling. When an error occurs at any step, the orchestrator must initiate a retry. While this seems simple in theory, it frequently results in a leaky abstraction, where logic that should ideally reside within domain services inadvertently shifts to the orchestrator. In contrast, our proposed approach allows for the management of retries and failure responses to be delegated to individual micro services, maintaining a cleaner separation of concerns.
Increasing responsiveness and scalability in Orchestration
Orchestration typically involves synchronous communication with downstream services, placing all such interactions on the critical path of the user experience. This can make the entire system seem slow from the user’s perspective. To address this, our approach shifts processing out of the critical path, significantly enhancing the system’s responsiveness. By doing so, we reduce the direct impact on user experience, leading to a perceptibly faster and more efficient system.
Typically, orchestration can result in reduced scalability due to its reliance on synchronous communication, where the system’s overall performance is limited to the sum of its individual services’ performances. However, by adopting eventual consistency in our approach, we can significantly enhance the scalability of our architecture, moving beyond the constraints of synchronous operations.
What are we building
We will develop a system for a town’s network of parking facilities. This system will manage parking sessions, including initiating and ending sessions, calculating parking fees, and processing payments.
Workflow:
- A vehicle owner starts a parking session upon entering a parking facility.
- The Session Management Service records the session details, including the spot number and entry time.
- When the session ends (the vehicle leaves), the Pricing Service calculates the fee based on the total time parked.
- The owner is charged when session is ended.
- All of the above is being Orchestrated by session Orchestration service
As illustrated in the diagram, the orchestration process involves the orchestrator initiating requests to each downstream service. Each service promptly acknowledges receipt of the request and then, upon completing its processing, notifies the orchestrator through a separate call. This separation of initiation and completion ensures that the services operate independently and efficiently.
State Management
state of workflow is managed in Session Orchestrator. I am using a library called Stateless to maintain state of workflow, good thing about this is that it makes state and state transition as first class citizen. For example, it will not allow wrong transition and throw an exception. You can do it yourself, but it is error prone.
Eventual Consistency
An important aspect to emphasize is that all calls in this orchestration are designed to be eventually consistent. This implies that handling errors and failures falls within the purview of each individual microservice. For instance, if the process to end a session encounters an error, the Session Service is responsible for retrying until it succeeds, after which it will inform the Orchestrator of the update. This approach is advantageous as it grants autonomy to each service, allowing the Orchestrator to remain streamlined and unburdened by complex error-handling logic.
Code
Services are written in C#. Few things worth mentioning:
- Stateless Library: This library is utilized to manage state transitions effectively. What stands out about Stateless is that it treats states and transitions as first-class citizens within the codebase, providing clear visibility and manageability of state within the services.
private void ConfigureStateMachine()
{
_stateMachine.Configure(WorkflowState.SESSIONS_STARTED)
.Permit(Trigger.StopSession, WorkflowState.SESSION_STOPPED)
.Permit(Trigger.FailSession, WorkflowState.SESSION_FAILED);
_stateMachine.Configure(WorkflowState.SESSION_STOPPED)
.Permit(Trigger.PriceSession, WorkflowState.SESSION_PRICED)
.Permit(Trigger.FailSession, WorkflowState.SESSION_FAILED);
_stateMachine.Configure(WorkflowState.SESSION_PRICED)
.Permit(Trigger.PaySession, WorkflowState.SESSION_PAID)
.Permit(Trigger.FailSession, WorkflowState.SESSION_FAILED);
_stateMachine.Configure(WorkflowState.SESSION_PAID)
.Permit(Trigger.CompleteSession, WorkflowState.SESSION_COMPLETED)
.Permit(Trigger.FailSession, WorkflowState.SESSION_FAILED);
}2. Jaeger: This is employed for distributed tracing, allowing you to track requests as they traverse through the various services. Jaeger is instrumental in diagnosing and monitoring the flow of requests and system behavior in a distributed setup.
3. MediatR: This library is used for event handling and dispatching tasks to background processes, facilitating eventual consistency. MediatR enables a clean separation between command sending and processing, which aligns well with the principles of event-driven architecture and helps in achieving decoupled service interactions.
The codebase, where these implementations are demonstrated, is provided for reference, offering insights into the practical application of these technologies within a microservices architecture.
Code: https://github.com/naveen-negi/FairyTaleSaga/tree/3eb5b3dd66c76924a4939a55b2178e30136c832c
Most beautiful part of this patterns is that compliexity is quite low; it includes the most convineit options (orchestrated, synchornicity) with the loosest restriction (eventual consistenct).
Overall system is also quite responsive because of async nature of communication. It also scales well because of eventual consistency.