← Back to blogs

The Unreliable Network: Why Two Generals Can Never Agree

State Machine Replication said order is everything. The network's reply is simple: you cannot guarantee delivery of that order. The Two Generals' Problem is not a metaphor — it is a proof that perfect agreement over a lossy link is impossible, no matter how many ACKs you add.

← Previously: State Machine Replication: Why Order Is the Only Thing That Matters

1. The Setup: A Valley, Two Hills, One Enemy

General A and General B must attack at dawn together or not at all. A lone attack means defeat. They communicate only via messengers who may be captured — the network drops packets with non-zero probability. Every message the generals send is an attempt to reach common knowledge: not just that both know the plan, but that both know that both know, ad infinitum.

In systems terms: two nodes, asynchronous unreliable links, crash-free but message-lossy. Can they commit a transaction atomically? The answer determines whether you can build a two-node strongly consistent database — and the answer is no.

2. Why ACKs Do Not Fix It

A sends "Attack at 06:00". B replies "ACK: 06:00". A replies "ACK the ACK". Each ACK raises confidence but never reaches certainty. Whoever sends the last message cannot know it arrived, so cannot safely attack. Remove that last message and you are back to the previous uncertainty.

A ──► "Attack at dawn" ──► B
A ◄── "ACK" ────────────── B   (did A get it? B doesn't know yet)
A ──► "ACK the ACK" ─────► B   (did B get it? A doesn't know yet)
...infinite regress...

TCP's three-way handshake does not solve this. TCP solves reliable delivery between two endpoints when both are live by giving up on exactly the guarantee we need: it can declare the connection broken instead of committing. Databases cannot "declare the partition broken" and walk away — money moved or it did not.

Interact below: Drive the generals' exchange yourself. Each click sends the next message in the chain. Toggle "Drop next packet" to swallow an acknowledgment and watch coordination collapse — even after multiple successful round trips, one loss reopens infinite doubt.

3. The Production Translation: Split-Brain

Replace generals with database replicas, messengers with TCP, and "attack" with "commit the write." A two-node primary/secondary that commits locally and ships asynchronously is two generals optimistic that the messenger arrived. When the link partitions after commit but before replication, the secondary promotes itself — now two primaries accept divergent writes. Reconciliation is not a merge; it is manual conflict resolution or silent data loss.

Primary (A) ──► "commit txn #42" ──► Replica (B)  [DROPPED]
Primary commits & acks client.
Partition heals: B never saw #42 → split-brain. Two histories exist.

Every "I have never seen split-brain in production" story is a story about a partition that has not happened yet, or a workload that never noticed the fork.

4. The Only Escape: Quorums

If you cannot guarantee delivery to a specific node, stop requiring it. Require a majority instead. In a 3-node cluster, committing when 2 of 3 have persisted means at least one survivor of any single failure carries the write. In a 5-node cluster, 3 of 5. The math is brutally simple:

• Quorum size = floor(n/2) + 1. Any two quorums intersect in at least one node.
• Overlap guarantees durability. No two majorities can commit different values at the same log index.
• Minority sacrifices availability. A 2-node system has no majority of 2 distinct from the whole — any partition makes it unavailable. That is not a bug. That is the proof made visible.

-- 3-node Raft/Paxos: commit needs 2 acks, not 2 specific acks
-- Client write succeeds when any 2 of {A,B,C} persist entry
-- Throughput = f(quorum latency p50), not f(slowest node)

Quorums also explain why gossip and "best-effort replication" feel resilient but are not consistent: they guarantee eventual delivery with high probability, not agreement on order. That is the gap SMR demands.

5. What Two Nodes Can Actually Do

Two nodes can provide durability with synchronous replication and a tiebreaker: a third witness that stores no data but votes. Witness-assisted consensus (etcd with a learner, PostgreSQL with Patroni + DCS) turns two data nodes into a three-voter cluster. The witness never holds the dataset, but it breaks the symmetry that makes two generals impossible.

Without a quorum, your options narrow to explicit trade-offs you document and staff for: last-writer-wins with vector clocks, CRDTs that restrict operations to commutative types, or accepting that partitions will fork and you will run repair jobs. Those are valid designs — they are just not strongly consistent.

Quorums quietly assume one more thing: that every replica agrees what "before" and "after" mean. Once you look closely at two machines whose clocks never tick in unison, that assumption cracks — and with it, the ordering a quorum is meant to protect. The mechanics of that crack, and the logical-clock alternative that doesn't trust wall time at all, are in The Illusion of Time: Why Your Timestamps Lie →