Distributed systems fundamentals
A distributed system adds independent failure, uncertain time, partial knowledge, and network cost. Correct architecture starts by rejecting the fallacies of distributed computing.
Core ideas and patterns
| Idea | What it solves | Cost, limit, or warning |
|---|---|---|
| Partial failure | Recognizes one component can fail while others run | Status becomes uncertain, not simply up/down |
| Timeout | Bounds waiting for remote work | Too short creates false failure; too long consumes capacity |
| Consistency model | Defines what observations are allowed | Weak guarantees move complexity to users and code |
| Consensus | Agrees on ordered state despite failures | Requires quorum and sacrifices availability under partition |
| Leader election | Assigns temporary authority | Split brain and fencing must be prevented |
| Logical clock/version | Orders causally related changes | Does not provide physical wall time |
| Quorum | Uses overlapping read/write sets for consistency | Latency and availability depend on quorum placement |
| Fencing token | Rejects stale lock holders | Every protected resource must enforce the token |
| Gossip | Spreads state without central coordination | Convergence is eventual and bandwidth is probabilistic |
How to apply it
State the required consistency per invariant. Define partitions, authority, ordering scope, clock assumptions, timeout and retry budgets, and behavior under partition. Prefer local transactions and explicit asynchronous coordination over pretending remote calls are local.
Evidence, not opinion
Use fault injection, clock skew, packet loss, process pause, duplicate/reorder tests, Jepsen-style history analysis where justified, and reconciliation invariants.
Small example
A lease alone cannot prevent an old worker writing after a pause. A monotonically increasing fencing token lets storage reject the stale writer even if it wakes later.
Feynman check
Explain the design to a new engineer without using the pattern names. State the problem, the forces that conflict, the chosen boundary or mechanism, what can fail, and the evidence that would prove the choice still works.