Observability, operability, SRE, and incident architecture
A system is operable when teams can understand state, control change, handle failure, recover data, and learn. Observability is designed into contracts and context propagation, not added after deployment.
Core ideas and patterns
| Idea | What it solves | Cost, limit, or warning |
|---|---|---|
| Logs | Record discrete contextual events | High volume, secrets, and inconsistent fields reduce value |
| Metrics | Aggregate numeric behavior over time | Cardinality and averages can hide incidents |
| Traces | Connect work across distributed boundaries | Sampling can miss rare critical paths |
| SLO/error budget | Balances reliability promises and change | Poor indicators optimize the wrong experience |
| Correlation/causation ID | Links related work and business intent | One trace ID may not cover long workflows |
| Runbook | Provides tested operational action | Stale prose can be dangerous |
| Feature flag | Separates release from exposure | Flags create state space and need retirement |
| Progressive delivery | Limits exposure while evaluating evidence | Bad metrics can promote bad releases |
| Post-incident learning | Improves system conditions after failure | Blame suppresses useful evidence |
How to apply it
Define user journeys and SLIs before dashboards. Propagate technical and business correlation across sync calls, messages, and jobs. Make state transitions and retries visible. Design safe control actions: drain, pause, replay, quarantine, fail over, roll back, and reconcile.
Evidence, not opinion
Prove alert precision, trace continuity, log redaction, dashboard usefulness, runbook execution, rollback time, restore, and incident roles. Use synthetic transactions and controlled failure.
Small example
An order has request ID, trace ID, order ID, saga ID, tenant ID, and message ID. Operators can follow one business outcome across minutes without searching by customer PII.
Feynman check
Explain the design to a new engineer without using the pattern names. State the problem, the forces that conflict, the chosen boundary or mechanism, what can fail, and the evidence that would prove the choice still works.