I’ve led a large-scale Medallion lakehouse programme on Azure Databricks and Unity Catalog. The technology was rarely the hard part. What made the difference was a small set of governance habits we agreed early and applied everywhere. Here are the ones I’d repeat on any programme.
1. Design the catalog layout before the first table
Unity Catalog uses a three-level namespace: catalog.schema.table. Decide what each level means and stick to it. I usually separate catalogs by environment and domain (for example prod_supply_chain) and use schemas for layers or subject areas. Then use workspace-catalog binding so a development workspace simply can’t write to a production catalog.
2. Grant to groups, never to individuals
Sync groups from Microsoft Entra ID at account level and grant privileges to those groups only. Make groups or service principals the owners of catalogs and schemas, and run production jobs as service principals. When someone changes team, you update a group membership instead of hunting down dozens of grants.
3. Prefer managed tables and govern external storage
Managed tables let Unity Catalog control the storage location and lifecycle, and they qualify for predictive optimisation, which can run maintenance such as OPTIMIZE and VACUUM for you. Where you do need external data, register it through storage credentials and external locations rather than handing out direct storage access. That way every read still goes through Unity Catalog permissions.
4. Use governed tags and ABAC for sensitive data at scale
Table-level row filters and column masks are fine for a handful of tables. Across hundreds of tables they become hard to keep consistent. Attribute-based access control (ABAC) lets you define a row filter or column mask policy once, based on governed tags such as pii, and apply it across a whole catalog or schema. Check the compute requirements first: ABAC needs serverless compute or Databricks Runtime 16.4 and above.
5. Turn on system tables and actually look at them
Unity Catalog captures lineage, and system tables expose audit logs, lineage and billing usage that you can query with SQL. I build a small governance dashboard on top of them that shows who accessed sensitive tables, which jobs cost the most, and which tables have no downstream consumers. It turns governance from a policy document into something you can check.
6. Share data, not extracts
When a partner or another platform needs data, use Delta Sharing instead of exporting files, and use Lakehouse Federation to query external databases under the same governance. Every CSV that leaves the platform is a copy you can no longer secure or audit.
7. Document tables where people will see them
Add comments to catalogs, tables and columns, and give every schema a named owner. Catalog Explorer shows these to analysts, and AI features such as Genie and your own agents rely on good descriptions to give sensible answers. Databricks can suggest AI-generated comments; review them before you accept them.
Wrapping up
Unity Catalog gives you the controls. Your naming, ownership and grant model decide whether those controls stay manageable after the hundredth table. Agree them early, write them down, and automate them wherever you can.
Leave a Reply