Reliability and resilience patterns
Resilience is the ability to preserve or recover required outcomes under disturbance. Patterns must be composed against a failure model; blindly stacking retries and redundancy can amplify failure.
Core ideas and patterns
| Idea | What it solves | Cost, limit, or warning |
|---|---|---|
| Retry with jitter | Handles transient failure | Retries multiply load and can repeat unsafe work |
| Circuit breaker | Stops calls likely to fail | Thresholds and half-open behavior need tuning |
| Bulkhead | Partitions capacity to contain exhaustion | Unused reserved capacity can reduce efficiency |
| Timeout/deadline | Bounds resource occupation | Nested timeouts must fit one end-to-end budget |
| Rate limiting | Protects scarce capacity and fairness | Limits need identity and priority semantics |
| Load shedding | Rejects low-priority work before collapse | Business degradation order must be agreed |
| Fallback | Provides a bounded alternate result | Stale or misleading fallbacks can be worse than failure |
| Health endpoint | Exposes readiness/liveness state | Shallow checks create false confidence |
| Redundancy | Survives component loss | Common dependencies defeat independence |
| Backup/restore | Recovers lost or corrupted state | A backup is unproven until restored |
How to apply it
Create a failure-mode table: trigger, scope, detection, containment, degraded behavior, recovery, data impact, and owner. Set an end-to-end deadline and allocate budgets. Retry only classified transient errors with idempotency and a retry budget. Design overload behavior before normal scaling.
Evidence, not opinion
Use SLOs/error budgets, fault injection, game days, restore drills, dependency maps, saturation alarms, reconciliation, and incident learning. Measure recovery time distribution, not the runbook estimate.
Small example
Checkout has a 1.5-second deadline. Inventory gets 400 ms with one jittered retry only for connection reset. Payment gets no automatic retry without an idempotency key. Recommendation load is shed first.
Feynman check
Explain the design to a new engineer without using the pattern names. State the problem, the forces that conflict, the chosen boundary or mechanism, what can fail, and the evidence that would prove the choice still works.