Most engineers understand idempotency as “don’t process the same request twice.” That understanding is close enough to pass an interview and far enough from the truth to produce real bugs in production. The actual problem idempotency solves is not duplication — it is uncertainty. In a distributed system, when you send a request and get a timeout, you do not know if the operation succeeded, failed, or is still in progress. You are not in a binary world of success and failure. You are in a world with a third state — unknown — and that third state is where most distributed system bugs live. Idempotency is the engineering discipline of making it safe to act when you do not know what already happened. Distributed state management is the discipline of maintaining correctness when different services disagree about the current state of the world, which, across a network boundary, they will.
The Level 1 post established that microservices and event-driven architectures replace local function calls with network calls, and that data consistency across service boundaries is a genuine problem. This post explains the mechanics of why that problem is hard and how idempotent design addresses it at the operation level.
The Third State: Why Retries Are Inevitable
A function call within a single process either returns a result or throws an exception. There is no ambiguity about whether it executed. A network call introduces a category of failure that does not exist locally: the caller’s connection times out, and the caller has no way to determine whether the remote service processed the request, failed before processing it, or processed it and the response was lost on the way back.
Consider a payment service. Your order service sends a request to charge a customer $50. The connection times out after 30 seconds. Three things could have happened: the payment service never received the request, the payment service received it and charged the customer but the response was dropped by a flaky load balancer, or the payment service received it and is still processing it because a downstream call to the card network is slow. The order service cannot distinguish between these cases. And it must do something — it cannot leave the customer staring at a spinner forever.
The standard response to this ambiguity is to retry the request. Retries are not optional in distributed systems; they are a fundamental mechanism for making progress in the face of transient failures. But retrying a non-idempotent charge operation means the customer might be charged $100 instead of $50. The retry that keeps your system available is the same retry that corrupts your data. Idempotent design resolves this tension.
What Makes an Operation Idempotent
An operation is idempotent if performing it multiple times produces the same result as performing it once. This is a property of the operation’s effect on system state, not its return value.
Some operations are naturally idempotent. Setting a user’s email to alice@example.com is idempotent — doing it three times leaves the same state as doing it once. Deleting a record with a specific ID is idempotent — the second delete is a no-op. An HTTP PUT that replaces a resource at a known URI is designed to be idempotent by the protocol’s semantics.
Other operations are naturally non-idempotent. Incrementing a counter, appending a row to a ledger, charging a credit card — these all produce additional effects on each execution. Retrying them naively produces incorrect state. These operations must be engineered into idempotent ones, and the mechanism for doing so is attaching a unique identity to the operation itself.
The Idempotency Key Pattern
The most widely used approach is the idempotency key: a unique identifier, typically a UUID, generated by the client and sent with the request. The server uses this key to detect and deduplicate retries.
The mechanical flow works like this. The client generates a key — say req-abc-123 — before making the request, and includes it as a header or parameter. When the server receives the request, it checks a store (usually a database table or cache) for that key. If the key does not exist, the server processes the request, stores the result keyed by req-abc-123, and returns the response. If the key already exists, the server skips processing entirely and returns the stored response from the first execution.
Client Server
| |
|-- POST /charge |
| Idempotency-Key: req-abc |
| |-- Check key store: not found
| |-- Process charge: $50
| |-- Store result for req-abc
| 200 OK ------|
| (response lost) |
| |
|-- POST /charge (retry) |
| Idempotency-Key: req-abc |
| |-- Check key store: found
| |-- Return stored result
| 200 OK ------|
The customer is charged exactly once. The client gets a response. The system is correct despite the retry.
Several mechanics are non-obvious here. First, the key must be generated by the client, not the server. The client is the one who does not know whether the first request succeeded — it needs to carry a stable identifier across retries so the server can correlate them. If the server generates the key, it is useless for deduplication because the client never received it.
Second, concurrent requests with the same key must be handled. If the client retries while the original request is still processing, you need either a lock on the idempotency key (so the second request blocks or returns a conflict status) or an optimistic check-and-set pattern. Without this, both requests could pass the “key not found” check and process simultaneously, defeating the purpose.
Third, the stored responses need a TTL. You cannot keep every idempotency key forever — the storage cost is unbounded. But the TTL must be long enough to cover the retry window. If a client retries after the key has been evicted, the operation will process again. In practice, TTLs of 24 to 72 hours cover most retry scenarios, but this depends on your system’s failure modes and retry policies.
Fourth, the idempotency key protects a specific operation at a specific scope. If your key is too broad (one key per user session), unrelated requests will collide. If it is too narrow (one key per HTTP connection), retries on a new connection will not be deduplicated. The key should be scoped to the logical operation: “this particular attempt to place this particular order.”
The Distributed State Problem
Idempotency handles the correctness of individual operations across retries. But distributed systems have a second, deeper problem: the state of your system is not in one place.
In a monolith backed by a single relational database, you can wrap a set of changes in a transaction. Deduct inventory and create an order and record a payment — all atomically. If any step fails, the whole thing rolls back. The database gives you ACID guarantees, and you never see a state where the order exists but the inventory was not deducted.
Once you split into services with their own databases, that transactional boundary is gone. The order service writes to its database. The inventory service writes to its database. These are two independent transactions on two independent systems. There is no way to atomically commit both. If the order is created and then the network call to deduct inventory fails, you have an inconsistent state: an order exists for a product that was never reserved.
Why Distributed Transactions Do Not Solve This
The classical solution to this problem is two-phase commit (2PC): a coordinator asks all participants to prepare, then asks them to commit. In theory, this guarantees atomicity across databases. In practice, 2PC is fragile and slow. If the coordinator crashes between the prepare and commit phases, all participants are stuck holding locks, unable to proceed or roll back, until the coordinator recovers. The protocol requires synchronous coordination between all participants, which means the operation is as slow as the slowest participant and as available as the least available participant. In a microservices environment with dozens of services, 2PC creates a bottleneck that negates most of the architectural benefits you decomposed to achieve.
This is why most distributed systems do not use 2PC across service boundaries. Instead, they accept a weaker consistency model.
Eventual Consistency as a Real Guarantee
Eventual consistency means that if no new updates are made to a piece of data, all nodes or services will eventually converge to the same value. This is not a vague aspiration — it is a specific guarantee with a specific implication: there is a window of time during which different parts of your system disagree about the current state.
In the order-and-inventory example, eventual consistency means: the order service records the order, then publishes an event (“Order Placed”). The inventory service consumes that event and deducts stock. Between the moment the order is recorded and the moment the inventory service processes the event, the two services are inconsistent. The order exists, but the inventory does not reflect it. If another request queries available inventory during this window, it gets a stale answer.
The length of this convergence window depends on your message infrastructure, consumer processing speed, and failure/retry behavior. Under normal conditions, it might be milliseconds. Under degraded conditions — a consumer is down, a message queue is backed up — it could be minutes or hours.
The engineering challenge of eventual consistency is not making things consistent eventually. That part is relatively straightforward if your messaging infrastructure is reliable. The hard part is designing your system to be correct during the window when it is inconsistent.
Making State Transitions Reliable Across Services
Two patterns dominate how production systems handle distributed state changes: the outbox pattern and sagas.
The Outbox Pattern
A common mistake is writing to your database and then publishing a message to a broker as two separate operations. If the database write succeeds and the message publish fails, the state has changed but no downstream service knows. If you publish first and the database write fails, downstream services react to an event that never actually happened.
The outbox pattern solves this by writing the event to an outbox table in the same database, in the same transaction as the business data change. A separate process (a poller or change data capture stream) reads the outbox table and publishes the events to the message broker. Because the business write and the outbox write are in the same transaction, they either both happen or neither does. The publishing step can safely retry because the downstream consumers are designed to be idempotent — which closes the loop on why idempotency is foundational to distributed state management.
Sagas
A saga is a sequence of local transactions across services, where each step has a corresponding compensating transaction that undoes its effect if a later step fails. Instead of one atomic transaction, you have a chain of smaller ones with explicit rollback logic.
For an order flow: the order service creates the order (step 1), the payment service charges the customer (step 2), the inventory service reserves stock (step 3). If step 3 fails — the item is out of stock — the saga executes compensations in reverse: refund the payment (compensate step 2), cancel the order (compensate step 1).
This works, but the compensating logic is real code that must be written, tested, and maintained. It handles partial failures explicitly rather than relying on a database to roll things back for you. And the system is in a partially-completed state between steps — the customer has been charged but inventory has not been reserved. Your system must handle queries and other operations that hit this intermediate state gracefully.
Tradeoffs and Failure Modes
Idempotency keys with incorrect scope are the most common implementation bug. A team adds idempotency support, but the key is derived from the request body rather than the logical intent. If the client changes the shipping address and retries, the key changes, and the retry creates a second order. Or worse: the key is derived from something too stable — like the user ID — and completely different operations get deduplicated against each other. The key must identify the specific attempt to perform a specific action, not the user, not the session, and not the payload.
Read-your-writes violations frustrate users and create support tickets. A user places an order. The order service writes the record and returns success. The user refreshes the page, and the query hits a read replica or a separate query service that has not yet received the event. The order appears to not exist. The user places the order again. Depending on your idempotency implementation, this either creates a duplicate or silently returns a cached response that confuses the UI. Solving this requires routing reads to the primary after writes (for database replication lag) or designing the client to optimistically show the result without re-querying.
Saga compensation failures are the most dangerous distributed state problem. What happens when the compensating transaction itself fails? You refund a payment, but the refund call times out. Now you have a charged customer, a cancelled order, and a pending refund that may or may not have processed. Sagas require their compensating transactions to be idempotent (so they can be retried) and monitored (so stuck compensations are detected and resolved, often manually). There is no framework that makes this fully automatic. At the edges, distributed systems require human intervention, and a well-designed system makes those edges visible rather than pretending they do not exist.
Treating eventual consistency as strong consistency is the subtlest and most pervasive failure mode. It manifests as code that writes to one service and immediately reads from another, assuming the write has propagated. It looks correct in development and staging (where latency is low and nothing fails) and breaks intermittently in production under load. The fix is never “make the propagation faster” — it is designing the interaction so correctness does not depend on propagation having already happened.
The Model to Carry Forward
In a single-process, single-database system, you can reason about operations as either succeeding or failing, and about state as being whatever the last committed transaction left it. In a distributed system, neither of these assumptions holds. Operations exist in three states from the caller’s perspective: confirmed succeeded, confirmed failed, and unknown. State exists in multiple places simultaneously, and those places will, during normal operation, disagree.
Idempotency is not a feature you add to an API — it is a design posture that acknowledges the third state. Every operation that crosses a network boundary should be safe to retry, because retries are not an edge case. They are a primary mechanism by which distributed systems make progress. Distributed state management is the discipline of designing for correctness during disagreement, not after resolution. If your system is only correct when all services are consistent, it is only correct some of the time.
The Level 1 post noted that microservices introduce distributed systems problems. The mechanics underneath that statement are the three-state problem, the loss of transactional boundaries, and the convergence window of eventual consistency. Understanding these mechanics is what lets you design systems that are correct by construction rather than correct by luck.
Key Takeaways
- In a distributed system, every network call has three possible outcomes from the caller’s perspective — success, failure, and unknown — and most correctness bugs come from ignoring the third.
- An operation is idempotent if performing it multiple times produces the same system state as performing it once; some operations are naturally idempotent (set, delete), others must be engineered to be (increment, charge, append).
- Idempotency keys must be generated by the client, scoped to the specific logical operation, and stored server-side with a TTL long enough to cover the full retry window.
- Concurrent requests with the same idempotency key require explicit handling — locking or conflict detection — or parallel execution will defeat deduplication.
- Two-phase commit provides atomicity across databases in theory, but its fragility under coordinator failure and its synchronous coordination cost make it impractical across microservice boundaries.
- Eventual consistency is not a vague promise — it is a specific guarantee with a specific cost: a convergence window during which different services disagree, and your system must be correct during that window, not just after it closes.
- The outbox pattern ensures atomic local writes and event publication by writing both in the same database transaction, then relaying events to the broker separately — and this only works because downstream consumers are idempotent.
- Saga compensating transactions can themselves fail, which means they must be idempotent, retryable, and monitored — there is no framework that eliminates the need for human intervention at the edges of distributed state recovery.