A data warehouse ingests raw events from production systems and transforms them into queryable datasets for analytics. The transformation pipeline has stages: raw ingestion, cleaning and normalization, business-level aggregation. Medallion architecture names these stages: bronze, silver, gold. The naming is less important than the principle: each layer has a contract about data quality, and data only flows forward.

Bronze: Raw and Immutable#

Bronze is append-only raw data exactly as it arrived from the source: JSON payloads from Kafka, CSV exports from operational databases, API responses. No transformation, no cleaning. Bronze data is often unstructured or semi-structured.

Bronze is immutable. You never delete or modify bronze records. This matters for reprocessing: if you discover a bug in your silver transformation 6 months later, you reprocess bronze to fix silver. If bronze were mutable or cleaned before storage, you’d have no authoritative source to reprocess from.

Retention: bronze is expensive (high volume, raw format). Common policy: keep bronze for 90 days, then archive to cold storage.

Silver: Cleaned and Normalized#

Silver transforms bronze into structured, validated records. Schema enforcement (reject malformed records or route to a quarantine table). Deduplication (remove duplicate events from at-least-once sources). PII masking (hash or tokenize email addresses, phone numbers before they reach analysts). Type normalization (timestamps to UTC, currencies to a single unit).

Silver records have a stable schema that downstream gold layers depend on. Schema changes in silver are breaking changes and require coordination.

graph TD A[Source: Kafka events, DB exports, API data] --> B[Bronze: raw append-only storage] B --> C[Silver transformation job: schema validation, dedup, PII masking] C --> D{Valid record?} D --> |Yes| E[Silver: clean structured records] D --> |No| F[Quarantine table: malformed or duplicate] E --> G[Gold transformation: business aggregations] G --> H[Gold: daily active users, revenue by region, funnel metrics] H --> I[BI tools, dashboards, ad-hoc SQL] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style I fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Gold: Business Aggregations#

Gold contains pre-computed business metrics: daily active users, revenue by region, conversion funnel by cohort, churn rate by plan. These are the tables BI tools and dashboards query directly.

Gold tables are wide and denormalized: joins are expensive at query time, so they happen in the transformation job. A gold table might combine data from 5 silver tables into one pre-joined record.

Gold is rebuilt from silver on a schedule (hourly, daily) or incrementally (process only new silver records since the last run). Incremental is faster but requires careful handling of late-arriving data: an event from 3 hours ago that arrives in bronze now must still flow into the correct gold partition.

Reprocessing#

The power of the three-layer model is reprocessing. Bug in your revenue calculation discovered 2 months later? Reprocess silver from bronze (bronze is intact). Bug in your silver dedup logic? Reprocess gold from corrected silver. Each layer can be reprocessed independently because the previous layer is immutable.

At Salesforce#

Salesforce’s analytics data pipeline for CRM usage metrics used a two-layer model (raw + curated) before moving to three layers. The original two-layer model applied PII masking in the same step as deduplication. When GDPR requirements added new PII field categories, reprocessing required re-ingesting raw data because the original bronze was already masked. After separating into three layers with bronze kept fully raw (with access control), PII mask updates required reprocessing only silver to gold, not re-ingesting raw data. Reprocessing time dropped from 14 hours to 2 hours for the full dataset.

What I’m Learning#

Medallion architecture is about immutability and reprocessability at each layer. Bronze’s immutability is what makes the whole pipeline auditable and correctable. The cost is storage at each layer, but the operational benefit of being able to reprocess any layer independently without re-ingesting source data is worth it at scale.

Have you had to reprocess a data pipeline due to a transformation bug, and did your architecture support it cleanly?