The Silent Failure Modes of Distributed Consensus

The Silent Failure Modes of Distributed Consensus

By Kevin Vance

Every backend engineer eventually learns that simplicity is not merely an aesthetic preference. It is an operational control. Splitting a working application into networked services replaces local function calls and database transactions with timeouts, retries, partial completion, duplicated messages, and state that can disagree across machines.

That trade can be worthwhile when a real boundary demands independent scale, isolation, ownership, or deployment. It is still a trade. A diagram that looks cleaner after being divided into boxes may create behavior that is harder to reason about when one of those boxes becomes slow rather than completely unavailable.

Start With the Local Invariant

The following TypeScript is illustrative application code. It keeps the read, decision, and write inside one database transaction so another worker cannot modify the same row halfway through the operation.

type Result = {
  entityId: string;
  state: string;
};

async function handleTransactionalOperation(entityId: string): Promise<Result> {
  return db.transaction(async (tx): Promise<Result> => {
    const record = await tx.queryOne(
      'SELECT id, state FROM entities WHERE id = $1 FOR UPDATE',
      [entityId]
    );

    const nextState = processEntity(record);
    await tx.execute(
      'UPDATE entities SET state = $2 WHERE id = $1',
      [entityId, nextState]
    );

    return { entityId, state: nextState };
  });
}

The important detail is not the library name. It is the invariant: either the transaction completes, or its writes roll back together. A caller receives one result rather than having to reconcile several independently committed steps.

What Changes Across a Network

Move the read and write into separate services and failure becomes ambiguous. A request can time out after the remote service has committed. Retrying may repeat a charge, notification, or state transition. A queue can deliver the same message more than once. Clocks can disagree, and a temporary partition can make two healthy processes hold incompatible views.

These problems have established controls: idempotency keys, unique constraints, transactional outboxes, bounded retries with jitter, deadlines propagated through the call chain, and reconciliation jobs. Each control adds code, storage, monitoring, and an operational procedure. The system should earn that cost through a demonstrated need.

Measure Before Dividing

Before adding a service boundary, capture the current latency distribution, query plans, resource limits, deployment bottlenecks, and failure history. Test whether an index, a smaller transaction, connection-pool tuning, or a background job solves the measured problem. Averages are insufficient; tail latency and behavior under dependency degradation matter.

If separation is justified, define who owns retries, how duplicate work is detected, which operation is authoritative, and how operators repair disagreement. Exercise those paths before production. A recovery plan that exists only in an architecture document has not yet been tested.

The Discipline of Subtraction

True engineering sophistication is often visible in what a team decides not to deploy. A single well-understood database is not automatically primitive, and a distributed system is not automatically scalable. The durable design is the smallest one that meets verified requirements while keeping failure bounded and recovery understandable.