Data architecture
The organisational and platform layer above the processing engines: who owns data, how it moves, and how anyone finds and trusts it.
| # | File | Covers |
|---|---|---|
| 01 | Data mesh and federated architecture | domain ownership, data as a product, data contracts, federated query, when mesh is the wrong answer |
| 02 | ETL, orchestration and dbt | ELT vs ETL, CDC and watermarks, idempotent backfills, Airflow/Dagster/Temporal, dbt models and tests, data quality |
| 03 | Catalogue, governance and lineage | discovery, trust and impact analysis, DataHub/OpenMetadata/Glue, column-level lineage, OpenLineage, governance that engineers implement |
Related
- Processing engines: ../01_spark_databricks/, ../03_polars/
- Storage formats: ../04_parquet/, ../05_delta_lake/
- Streaming: ../../10_message_queues/kafka/, ../../10_message_queues/stream_processing/
- Regulated-data provenance: ../../31_healthcare_regulated/03_gxp_and_validation.md