Concurrency control comes in two kinds, and most distributed-systems failures are a misassignment between them. Let’s look at where to put the single-writer guarantee, what each placement costs, and why the cheapest one has no failure mode at all.
There are two mechanisms for keeping two things from stepping on each other, and they are not variations on a theme. Serialization prevents concurrent access: a mailbox, a mutex, a lease, a leader election, a lock. Arbitration permits it and rejects the loser: a compare-and-swap, an expected stream revision, a unique constraint, a state machine that refuses a transition it has already performed.
Every real system uses both. What matters is which one you have made responsible for correctness. This is because the two have an asymmetry that does not show up until you are running more than one copy of your process.
Arbitration vs. Serialization
Arbitration is total. An expected-version check holds during a network partition. It holds when your coordination service is unreachable. It holds when a code path forgets to take the lock, when an admin runs a script against production, and when somebody adds a second service that writes the same table two years from now. You can’t bypass it because it is evaluated at the moment of the write by the thing doing the writing.
Serialization is conditional. It holds when every writer agrees to participate, and the thing handing out permission is reachable and correct. Those are real conditions, and each one is a way for the guarantee to evaporate quietly.
Put correctness on the arbiter. Put performance on the serializer.
Get that assignment backwards, and the system passes every test, works at one replica, and then fails in production in a way that reads as data corruption rather than contention — because by the time you see it, the damage is a committed history, not a stuck request.
What follows is that principle applied to an event-sourced system: a NestJS CQRS backend on KurrentDB and Postgres, running multiple replicas. The interesting decisions are all the same decision, asked four times.
Where Serialization Can Live
Serializing writes per entity is the classic requirement in event sourcing, because two writers appending to one stream produce a history no fold can interpret. There are only a few places to put that serializer, and they differ in ways worth having in front of you before you pick one.
| Placement | Cost per command | Guarantee scope | Evaporates when |
|---|---|---|---|
| In-process mailbox (actor, queue, mutex) | Pointer enqueue | One process | You run a second process |
| Sharded mailbox (cluster sharding) | Enqueue, plus a hop if the shard is remote | Cluster, at most one live entity | Partition without split-brain resolution; briefly, during rebalance |
| External lease (etcd, ZooKeeper, singleton) | Renewal traffic; an election on failover | Cluster, with a liveness hole | Holder is wedged but not dead, so nothing runs and nothing alerts |
| Database lock (advisory or row) | A round trip, on every command | Every writer that opts in | Database is slow or unreachable; a writer skips it |
| None (arbitration only) | Zero, plus retries under contention | Universal | Never. This is the one that cannot evaporate. |
Node lands you in the first row by default and disguises it as the fifth. One thread per process is a genuine serializer, so a read-modify-write across await points really is atomic — inside that process. It is the single-writer guarantee with a scope of one pod. And, scope is exactly the property that stops being free the moment there are two.
Our Choice
We took the database row. Every pod already shares a Postgres, so an advisory lock keyed on the entity ID is a cluster-wide FIFO queue with no new infrastructure. There’s also no membership protocol to operate.
// CommandLockService — serialize commands per entity id, cluster-wide. async withEntityLock<T>(entityId: string, operation: (tx: DrizzleTransaction) => Promise<T>) { const lockKey = this.hashEntityId(entityId); return this.drizzle.getDb().transaction(async (tx) => { await tx.execute(sql.raw(`SET LOCAL lock_timeout = '${this.lockTimeoutMs}ms'`)); await tx.execute(sql`SELECT pg_advisory_xact_lock(${lockKey})`); return operation(tx); }); }Two properties of this placement are worth stating because they are properties of the technique rather than of any codebase. Advisory locks are scoped to a session, and a connection pool is not a session. The protected operation has to be pinned to the connection that took the lock, which is what the surrounding transaction is for. And the key space is numeric, so string IDs get hashed, and unrelated entities occasionally collide onto one key and queue behind each other. Neither costs correctness. Both cost throughput, and only under load.
This is fine, because the lock is not what makes this correct. Behind it, every append carries the revision it was computed from. The store rejects it if the stream has moved. On rejection, the attempt is discarded and re-driven from a fresh read. There is no merge and no side effects, since events are only published after a successful write. That check is the arbiter. The lock exists so that it rarely has to fire.
Assign it that way and the lock is allowed to be approximate — allowed to time out, allowed to collide, allowed to be skipped by a code path that does not know about it. Assign it the other way, and you find yourself tuning lock timeouts to protect data integrity, which is a position with no good moves.
Arbitration Needs an Address
Here is the constraint that makes arbitration harder than it looks: every arbitration mechanism keys on an identity. Expected-version keys on a stream name. A unique constraint keys on a tuple. Compare-and-swap keys on a cell. The arbiter can only reject the loser of a race it can see, and it can only see races between operations that name the same thing.
Where It Usually Fails
So arbitration is exactly as good as your naming, and naming is where it usually fails. Consider a group identified by a natural key — an organization plus a normalized set of device serial numbers. Two registrations naming the same devices are, by definition, the same group. If the code looks up an existing group and mints a randomUUID() when it misses, then two concurrent registrations produce two IDs, hash to two lock keys, and write two streams. Both the serializer and the arbiter perform flawlessly, on two entities the system has no reason to believe are one.
The fix is not more coordination. It is to stop needing any:
Deriving an entity’s id from its natural key converts a coordination problem into a naming problem, and naming problems are solvable without coordination.
This is the same move as content-addressable storage, as idempotency keys on a payments API, as deterministic replica ids in a CRDT. Two actors who never communicate agree on an identifier because they compute it from the same inputs, and everything downstream that keys on identity — locks, streams, constraints — starts working without any of it knowing why.
// Never change this value: it would re-key every device group stream. const DEVICE_GROUP_ID_NAMESPACE = 'f2a7c1e8-4d36-4b90-9c17-5e8b3a0d76f4'; // WHY: `uniqueSerials` sorts with a bare localeCompare, which depends on the // runtime's default locale. Fine for its own callers, but this ordering is baked // into a permanent entity id, so it must be locale-independent. const compareByCodePoint = (left: string, right: string) => left < right ? -1 : left > right ? 1 : 0; export const deviceGroupEntityId = ( organizationId: string, serialNumbers: readonly string[], ): string => { const normalized = uniqueSerials(serialNumbers).slice().sort(compareByCodePoint); // WHY: JSON-encode rather than join on a separator. The set has variable arity, // so a plain join lets one serial containing the separator produce the same string // as a multi-serial set (['A,B'] vs ['A', 'B']) and collide onto one group id. // JSON encoding is injective for an array of strings. return uuidv5(`${organizationId}:${JSON.stringify(normalized)}`, DEVICE_GROUP_ID_NAMESPACE); };The technique has a price, and it is not the one people expect. It is not the hashing, and it is not the namespace constant. It is this: a derivation function is a schema.
A Technique and a Price
Every ID it has ever produced is already in the event store, under a stream name that cannot be renamed, referenced by rows that were written months ago. There is no migration for a derivation change short of rewriting history, so the function is frozen on first write, and every property it needs must be true from the beginning:
- Deterministic across runtimes. A sort that consults the default locale produces different orderings on different hosts, so two pods compute two ids for one input set. Collation is a configuration detail everywhere else in your system and a correctness property here.
- Injective. The encoding must map distinct inputs to distinct strings. Joining a variable-arity set on a separator does not:
['A,B']and['A', 'B']collapse to the same key. So, two genuinely different groups become one entity, and no arbiter downstream can tell that anything went wrong — they agree, which is precisely the failure. - Stable under normalization drift. The normalizer is now part of the schema too. Trimming, case folding, and deduplication rules are frozen alongside the function that calls them.
The failure mode of getting this wrong is worse than the race it replaced, because a duplicate entity announces itself and a collided one does not. Which is the argument for treating the derivation as a reviewed, separately tested artifact with a comment explaining each property, rather than an expression inlined at a call site.
There is also a design consequence that arrives with the technique. Collapsing two would-be entities into one makes the creating command’s payload a merge. Each racer knew about part of the state, one of them wins, and the winner’s view is missing whatever the loser knew. Deduplication moves the work from “prevent the second create” to “reconcile what the second create was carrying.” That reconciliation is not free.
Exactly-Once Is a Commit Boundary, Not a Delivery Mode
The read side asks the same question in different clothes. Projections consume persistent subscriptions as competing consumers. Delivery is at-least-once, and no configuration makes that stop being true. A pod can apply an event, die before acknowledging, and the event returns to whichever pod inherits the stream.
Not Available
“Exactly-once delivery” is not available and never was. What is available is this identity, which is worth stating precisely because it tells you exactly which effects can have it:
At-least-once delivery, plus an atomic commit of (effect, dedup marker), equals exactly-once effect.
Everything follows from the word atomic. The offset and the thing it certifies have to commit or fail together, which means they have to live inside the same transactional boundary. For a projection into the same Postgres, that is four lines:
// AbstractDbProjection — the offset commits inside the handler's transaction. protected async executeEvent(event: TEvent, context: EventContext): Promise<void> { await this.drizzle.runInTransaction(async () => { await this.handleEvent(event); await this.projectionOffsetRepo.setLastRevision( context.projectionId, context.entityId, context.revision, ); }); }Separate those two writes, and you own a window in which the read model advanced, and the offset did not, and redelivery applies the event twice. Nothing else about the subscription changes. The guarantee lives entirely in the fact that both statements are inside one BEGIN.
Read the identity in the other direction, and it partitions your side effects for you. An effect that cannot join the offset’s transaction cannot be exactly-once, ever, by any amount of retry logic. Sending mail, writing to object storage, calling a third-party API — all categorically outside the boundary. That is not an engineering shortfall to be fixed later. It is a property of where the effect lives, and it is load-bearing enough that in this system it shows up in the type hierarchy.
Two Base Classes
Every projection extends one of two base classes. They share a subscription, an idempotency check, and a handleEvent hook that subclasses implement. The only thing that differs is what the base class does with a failure, and that difference is decided entirely by whether the effect can sit inside the offset’s transaction.
AbstractDbProjection is the one above, and it covers anything whose output is a row in the same Postgres. It wraps handler and offset in one transaction and lets exceptions propagate. The transaction rolls back, the event is NACKed, and the store redelivers it with backoff indefinitely. A failing event therefore blocks every event behind it on that stream. That is intended rather than tolerated — a read model with a hole in it is worse than one that is merely behind, because skipping event N applies event N+1 to a state that never existed.
AbstractAsyncProjection covers everything outside the boundary. It cannot use that policy, since blocking a stream indefinitely over one failed notification is an outage wearing consistency as a costume. So it retries internally, writes the failure to an errors table, and then advances the offset regardless of how the attempt went:
// AbstractAsyncProjection — the effect cannot join the offset's transaction. protected async executeEvent(event: TEvent, context: EventContext): Promise<void> { try { await this.retryWithBackoff(() => this.handleEvent(event)); } catch (error) { await this.saveErrorState(event, error); // async_projection_errors } // Always update the offset — even on failure. The error is tracked separately, // and we do not want the subscription replaying this event forever. await this.projectionOffsetRepo.setLastRevision( context.projectionId, context.entityId, context.revision, ); }This reduces the choice for anyone writing a new projection to one question asked before picking a base class: does this effect commit in the same transaction as the offset? If it does, extend AbstractDbProjection and exactly-once comes for free. If it does not, extend AbstractAsyncProjection and understand that the final statement of that method is a debt.
It is a debt in the precise sense: the offset moves past an event whose effect did not land, the subscription will never revisit it, and the divergence is permanent unless something else goes looking. Choosing to advance past failure is choosing to owe a reconciler — a periodic pass that recomputes what should exist and repairs the difference. It belongs in the same commit as the projection, because a failure mode with no detector is indistinguishable from no failures at all.
What Deserves a Serializer at All
With the assignment rule in hand, most of the remaining questions answer themselves, and the answer is usually “nothing.”
Entity writes get a serializer, because contention on a single entity is common and the arbiter’s rejection path is expensive. A rejected append means a wasted read, a wasted fold, and a retry. Paying a round trip to avoid that is a good trade under contention and a bad one without it, which is a tuning decision, and tuning decisions are allowed to be wrong.
Scheduled work gets nothing. Every replica runs the sweep; the reflex to elect a leader is worth resisting, because a lease introduces a liveness failure the duplicate-work version does not have. A holder that is wedged but not dead means the job stops entirely. Plus, it alerts on nothing, since a singleton that is not running looks identical to a singleton with no work to do. Trading a detectable inefficiency for an undetectable stall is the wrong direction on any job that can be made re-drivable.
A Query
What replaces it is a query that knows what it is:
// IMPORTANT: every backend pod runs this scheduler; row locks with SKIP LOCKED are // only a short-lived work distribution hint. The event-sourced aggregate remains the // duplicate-finalize guard once commands run outside this transaction. // // This query finds work; it does not gate it. const rows = await tx .select({ id: batches.id }) .from(batches) .where(and(eq(batches.status, BatchStatus.Pending), lte(batches.holdExpiresAt, options.now))) .orderBy(asc(batches.holdExpiresAt)) .limit(options.limit) .for('update', { skipLocked: true });SKIP LOCKED is a serializer with a lifetime of one transaction, and the command it dispatches runs after that transaction ends. It stops two pods claiming the same rows in the same instant, which saves real work, and it does nothing about the same row a second later. The arbiter is the aggregate’s state machine, which accepts the command from one state and refuses it from every other. Writing that distinction at the call site is not documentation courtesy: SKIP LOCKED reads like a gate and behaves like a hint, and the next person will believe whichever one the code lets them believe.
Once duplicates are the design rather than the defect, the caller’s error path carries the weight. Losing a race and failing outright arrive as the same exception and must not be reported the same way — so the recovery re-reads the entity, treats the rejection as success only if the state is the one the command was trying to reach, and rethrows otherwise. Two outcomes, and only the one that did the work gets counted. That reads like bookkeeping and is actually alerting policy, because a system that reports “someone else already did it” as a failure trains its team to ignore that alert inside a week.
The narrowness matters as much as the forgiveness. A catch block that treats every conflict as benign converts a race into silent data loss, which is strictly worse than the crash it replaced. And the re-read has to go to the entity rather than the read model — the read model is written by an independent subscription and can lag the very event being acted on. Anything answering “did this actually happen” has to ask the arbiter, not a projection of the arbiter.
The Conservation Law
Serialization can be moved but not removed. Push it out of the process, and it becomes a network round trip. Push it into a coordination service, and it becomes an operational dependency with an election protocol. Delete it entirely, and it reappears as retries under contention. There is no placement where it costs nothing. This is why the placement is worth choosing deliberately instead of inheriting it from whichever framework you happen to be holding.
Arbitration is the part that does scale for free. It is evaluated at the write, by the writer, against state that is already there — no protocol, no quorum, no lease, nothing to keep alive. It is also the only mechanism in this article that survives everything: the partition, the outage, the forgotten lock, the script somebody runs at two in the morning.
So the question to ask of any concurrency control is not whether it works. It is which of the two things it is. If the answer is a serializer, it is tuning, and it is allowed to fail. If the answer is an arbiter, it had better be reachable from every path that writes, and it had better key on a name that two machines who have never spoken will both compute identically.
Node’s single thread is a real guarantee with a scope, and the scope is one process. Everything above is the cost of deciding, on purpose, how far past that scope you actually need it to reach.