The transition to open lakehouse architecture has revolutionized analytics, allowing seamless access across diverse engines and tools. Despite this flexibility, organizations have struggled to maintain consistent and fine-grained data governance as data moves from proprietary systems into open formats like Delta Lake and Apache Iceberg. Databricks’ latest Unity Catalog advancements promise a breakthrough: unified governance that doesn’t compromise security, scalability, or openness.
Challenges of Governing Multi-Engine Data Environments
With analytics engines such as Spark, Trino, and DuckDB accessing shared data, ensuring secure and consistent access control has been daunting. Security teams have historically faced three problematic choices:
- Duplicating access policies across various systems, creating risk and operational burden
- Maintaining filtered views or copies of data, resulting in duplication and inefficiency
- Granting overly broad access, undermining data privacy and compliance
While open lakehouse formats provide flexibility, organizations have demanded more: unified, fine-grained governance that transcends the boundaries of individual tools and platforms.
Unity Catalog: Centralized Policies, Universal Enforcement
The Unity Catalog introduces a single, scalable security model for attribute-based access control (ABAC) across all engines, both within and outside of Databricks. Administrators can set row filters and column masks once, using governed tags and user attributes, and see those rules enforced wherever the data is accessed. This approach eliminates the need for custom policy implementations or redundant data copies outside Databricks.
Server-side filtered scan plans make this possible. When an external tool requests data, Unity Catalog evaluates security policies and returns only the authorized data. This keeps sensitive information protected and governance consistently enforced, without manual intervention.
Expanding Fine-Grained Controls to External Engines
Traditionally, fine-grained controls like row-level filtering or personally identifiable information (PII) masking were limited to Databricks’ compute environment. External engines couldn’t natively support these detailed policies. With the new Unity Catalog preview, fine-grained controls now extend to all engines, enabled by the Iceberg REST catalog APIs. This delivers:
- Consistent ABAC enforcement in tools lacking native governance (e.g., DuckDB, Python, pandas)
- Scalable, efficient policy application without added complexity or latency
- A dynamic policy layer responsive to user entitlements and context
Centralized enforcement ensures policies are applied server-side before any data reaches external consumers, so sensitive information is always protected.
Open Standards as the Foundation for Unified Governance
Unity Catalog’s enforcement leverages the Iceberg REST catalog protocol, an emerging open standard. When an engine requests a scan, Unity Catalog applies all relevant policies and delivers a filtered plan, restricting data access to what each user is allowed to see. Databricks’ optimized serverless infrastructure handles this efficiently, balancing security and performance.
Databricks is collaborating with the open-source community to define shared policy standards, paving the way for an ecosystem where engines can natively enforce governance. This forward-thinking approach positions the lakehouse for truly enterprise-grade data security and interoperability.
The Open Lakehouse: Unifying Access, Storage, and Governance
This evolution unlocks the full potential of open lakehouse architecture: open storage, open table formats, open APIs, and unified governance. Enterprises can confidently use their tool of choice, assured that policies are defined once and enforced everywhere. This not only reduces security overhead but also promotes true interoperability and compliance regardless of where or how data is accessed.
What’s Next for Unified Data Governance
Databricks is onboarding select customers to preview these new fine-grained access controls for external engines, with broader support planned as more engines adopt the Iceberg REST catalog APIs. This marks a pivotal step toward a fully open, governed, and interoperable data ecosystem - making unified, fine-grained governance a reality for the modern enterprise.
Source:Databricks Blog
Achieving Unified Data Governance in the Open Lakehouse Era