← Back to blogs

Failure Models: The Crash, the Recovery, and the Liar

Not all failures are equal. A node that stops is an inconvenience. A node that recovers with amnesia is a protocol bug. A node that lies differently to each peer is an adversary. Your replication factor — and your choice of Raft vs. PBFT — is decided the moment you pick which of these you must tolerate.

← Previously: The Illusion of Time: Why Your Timestamps Lie

1. The Hierarchy That Dictates Cost

Distributed systems classify failures from weakest to strongest:

• Crash-Stop (fail-stop): Node halts and never returns. Simplest model; disk is assumed durable or irrelevant. Rarely true in production — machines reboot.
• Crash-Recovery: Node crashes, loses volatile state, restarts from stable storage. Dominant enterprise model. The trap is amnesia: a node that forgot it already voted can double-vote unless the vote was persisted to WAL before ack.
• Omission / Partition: Node is live but messages are dropped or delayed. Raft and Paxos treat this as a crash from the quorum's perspective — same recovery path.
• Byzantine (arbitrary): Node may send conflicting messages, equivocate, or collude. A corrupted firmware, a compromised supply chain, or a bit-flip that passes checksums. 3f+1 to tolerate f Byzantine nodes vs. 2f+1 for crash tolerance — a steep price for a threat most internal systems do not face.

CFT (crash):  f faults → 2f+1 nodes (quorum f+1). Raft, Paxos, Zab.
BFT (Byzantine): f faults → 3f+1 nodes (quorum 2f+1). PBFT, HotStuff, Tendermint.
Cost of distrust: for f=1, CFT needs 3 nodes; BFT needs 4.

2. Why Enterprise Databases Choose CFT

Inside a single trust boundary — your VPC, your operator, your kernel — Byzantine behavior is vanishingly rare compared to crashes, OOM kills, and network partitions. Paying 33% more nodes plus cryptographic verification and view-change complexity for BFT is not justified when you can instead fence misbehaving nodes.

Fencing is how CFT systems deal with the "zombie leader" — a partitioned former leader that still thinks it is authoritative. Techniques: epoch numbers (ballot/term) that outrank stale leaders, STONITH (Shoot The Other Node In The Head) via the control plane, and lease expiry enforced by followers. The stale leader's writes are rejected because its term is obsolete, even though it is not malicious — just outdated.

Try it below: A 5-node cluster tolerates f=2 crashes (CFT) but only f=1 Byzantine fault. Crash nodes and watch the quorum stay green. Switch a node to Byzantine — it sends a teal value to nodes {A,B} and amber to {C,D} — and watch how CFT quorum (3) can be tricked while BFT quorum (4) would still require a liar to convince a supermajority.

3. Crash-Recovery: The Amnesia Bug

The subtlest crash failure is not the crash — it is the recovery. A node that replays from disk must have persisted every promise it made before acknowledging. Classical example: Paxos acceptor that voted for value v at ballot 7, crashed before fsync, rebooted, voted for w≠v at ballot 8. The protocol promises "never vote twice at different values for the same slot" — violated by a missing fsync().

-- Safe acceptor (crash-recovery):
-- 1. WAL write: ballot=7, vote=v, promised=maxBallot
-- 2. fsync() before
-- 3. send Accepted to leader
-- On recovery: replay WAL → state restored before serving.

Production consequence: every Raft persistent state (currentTerm, votedFor, log[]) must be on stable storage before RPC reply. etcd, CockroachDB, and PostgreSQL patroni all fsync the WAL on the critical path. The latency cost is the price of crash-recovery correctness — batch and pipeline to amortize, but never skip.

4. When You Actually Need Byzantine Tolerance

Cross-organization replication, public ledgers, or hardware you genuinely do not trust. A cloud provider that hosts your replicas cannot be assumed crash-only if the threat model includes a compromised hypervisor or a network adversary that forges messages. BFT protocols add two ingredients:

• Quorum intersection at 2f+1. Any two BFT quorums overlap in at least f+1 honest nodes, so a liar cannot assemble a quorum alone.
• Cryptographic authentication. Every message signed; equivocation is detectable because the same node signed two different values at the same sequence number.

Operational rule of thumb: if you control the hosts, choose CFT and invest the saved capacity in better fencing, faster failure detection (phi-accrual), and disciplined fsync. If you do not control the hosts, assume Byzantine and budget 3f+1.

5. The Failure Model You Forgot: The Operator

The most common "Byzantine" fault in enterprise systems is a human running UPDATE users SET admin=true on the primary and expecting replication to fix the semantics. Replication faithfully propagates the mistake to every replica. SMR guarantees convergence on the same log — not that the log was wise.

With order, quorums, clocks, and failure definitions in place, one constraint still applies even when everyone is honest and perfectly synchronized: you can only optimize for two things at once. That trade-off — whether you pay in availability or latency — is mapped in The Boundaries of Physics: CAP and the PACELC You Actually Live With →