Data architecture, ownership, consistency, and analytics
Data architecture defines meaning, ownership, lifecycle, quality, access, lineage, placement, movement, and consistency. Database selection comes after workload and invariant analysis.
Core ideas and patterns
| Idea | What it solves | Cost, limit, or warning |
|---|---|---|
| System of record | Names authoritative state for a business fact | Different fields may have different authorities |
| Polyglot persistence | Matches stores to workload needs | Multiplies skills, operations, and consistency models |
| Data warehouse | Integrates historical structured analytics | Batch freshness and central modeling can limit speed |
| Data lake/lakehouse | Stores broad raw and curated analytical data | Governance and quality are not automatic |
| Data mesh | Applies domain ownership and product thinking to analytical data | Platform and governance maturity are prerequisites |
| Materialized view | Precomputes read results | Refresh lag and rebuild policy matter |
| Change data capture | Streams committed database changes | Database logs expose physical rather than domain semantics |
| Sharding/partitioning | Distributes data and load | Cross-partition operations become expensive |
| Schema evolution | Changes structure without breaking consumers | Backward/forward compatibility needs discipline |
How to apply it
Classify data and assign semantic owner, steward, system of record, retention, residency, access policy, quality rules, lineage, recovery objective, and deletion process. Model transactions and queries. Choose partition keys from access and growth, not convenience. Version contracts and migrate expand-contract.
Evidence, not opinion
Measure freshness, completeness, accuracy, uniqueness, reconciliation, lineage coverage, restore results, query latency, skew, and unauthorized access. Test deletion across replicas, caches, indexes, and analytics.
Small example
Customer contact preference is authoritative in the preference service, copied to campaign projections with version and timestamp, and reconciled daily. Campaign cannot overwrite the source record.
Feynman check
Explain the design to a new engineer without using the pattern names. State the problem, the forces that conflict, the chosen boundary or mechanism, what can fail, and the evidence that would prove the choice still works.