Almost every lakehouse I see uses the medallion pattern, and almost every one draws the same three boxes. The difference between a platform that runs quietly and one that needs constant firefighting is in the details behind those boxes. These are the design rules I apply, whether the platform is Databricks or Microsoft Fabric.
1. Write a contract for each layer
Agree in writing what each layer promises. Bronze: data as received, append-only, with ingestion metadata such as source file and load timestamp. Silver: typed, cleaned, deduplicated and conformed to shared keys. Gold: modelled for a named consumer, such as a semantic model, a data product or an AI agent. When nobody can say what a layer guarantees, logic leaks into the wrong place.
2. Make every load idempotent
Running the same load twice for the same window should give exactly the same result. In practice that means MERGE on business keys instead of blind appends into silver, deterministic deduplication rules, and overwriting whole partitions or windows when you reprocess. Idempotency is what makes reruns, backfills and failure recovery boring, and boring is good.
3. Track incremental state explicitly
Don’t rely on “load everything since yesterday” logic based on the clock. Use streaming checkpoints with Auto Loader, Delta change data feed, or a simple watermark table that records what was processed. Explicit state lets you answer the question “has this record been loaded?” without guessing.
4. Put the layer in the namespace, not the table name
I keep layers as catalogs or schemas (for example silver.orders) rather than table names like orders_silver_v2_final. Pick snake_case, use dim_ and fact_ prefixes in gold, and keep column suffixes consistent, such as _id, _date and _ts. Boring, predictable names save hours for every new engineer and every AI tool that reads your metadata.
5. Put quality gates between layers
Validate data on the way into silver and gold, not after a report looks wrong. In Lakeflow pipelines (formerly Delta Live Tables), expectations let you choose to warn, drop or fail on bad records. Whatever tool you use, send rejected rows to a quarantine table with the reason attached, so they can be fixed and reprocessed.
6. Be tolerant in bronze, strict in silver
Source systems change without telling you. Let bronze absorb that: keep unexpected fields, for example in a rescued data column, rather than failing the ingest. Silver should enforce an agreed schema, and a schema change there should be a reviewed code change, not a surprise.
7. Don’t let gold become a dumping ground
Gold tables should exist because someone uses them. Tie each one to a consumer and an owner, and retire tables nobody reads. A smaller, well-described gold layer is easier to secure, cheaper to run and much better for Direct Lake models and AI agents.
Wrapping up
The medallion pattern isn’t a framework; it’s a set of promises. Write those promises down, make loads repeatable, and enforce quality at the boundaries. The three boxes will then mean the same thing to everyone.
Leave a Reply