The Open Data Lakehouse: A Data Platform Without Lock-in

What is an open data lakehouse?

An open data lakehouse is a lakehouse architecture in which data is held in open file and table formats and different compute engines work against the same data layer.

• Changing a query engine should not require moving the data
• Changing a processing technology should not require rebuilding the whole platform
• Changing cloud provider should not force the data architecture to be redesigned

The principle behind it comes down to three things:

Data belongs to the organization, not to the technology that processes it.

What is vendor lock-in?

Vendor lock-in is what happens when changing the platform or technology an organization uses becomes technically, operationally or economically very hard. In data platforms it can form at several levels.

Vendor lock-in is not a wrong choice in every case. Managed, integrated platforms can give some organizations operational ease and a fast start.

What matters is whether the organization chose that dependency knowingly, and whether it knows what changing course would later cost.

Open file formats

Open data formats are one of the lowest layers of an open lakehouse. Apache Parquet is among the open, column-oriented file formats widely used for analytic workloads.

Holding data in open formats lets the same data files be used by different technologies.

But using Parquet files is not on its own enough to make a lakehouse. At enterprise scale, tables' metadata, schema, partitioning, transactions and snapshot state need managing too — which is where open table formats come in.

Apache Iceberg and open table formats

Apache Iceberg is an open table format built for large analytic datasets. It offers schema evolution, partition evolution, hidden partitioning and snapshot-based table state, while letting the data be used by different compute engines.

With the right integrations, the same Iceberg table can be used by:

• Apache Spark
• Trino
• Flink
• Other analytical technologies

The table format is therefore separable from the compute engine, and that separation is one of the founding principles of an open lakehouse.

A compute engine does not own the data

In traditional data platforms, storage and compute may be parts of the same system. In the open lakehouse approach the two layers are separated:

One consequence matters more than the rest: the compute technology can change while the data layer stays.

An organization using one query engine for certain SQL workloads today may want a different technology in future. If the data is held in open table formats, such a change need not mean moving all of the enterprise data out of one proprietary system and into another.

One platform does not mean one technology

The idea of an enterprise platform is sometimes read as requiring every technology layer to come from the same vendor. Kauzas takes a different view.

A data platform can use one technology for storage, another for the table format, another for the catalog, another for SQL queries and another for data processing.

The platform's job is not to replace all of these with its own proprietary equivalents, but to make an enterprise data platform out of technologies that work together and can be governed together.

One platform does not mean one technology.

Replaceable technology layers

One of the Kauzas Data Platform's founding architectural principles is that its technology layers are independent and replaceable.

Each layer is separated from the others as far as open interfaces and standards allow.

The question behind a technology choice then becomes “which technology suits this workload?” rather than “which technology does Kauzas require?”

The layered Data & AI architecture →

What happens when a technology changes?

Suppose the technology an organization uses for SQL queries stops meeting its needs. In a tightly coupled architecture, that change can require:

• Moving the data
• Rebuilding the metadata
• Changing the pipelines
• Rewriting the applications

In an open, separated architecture the aim is this: storage stays, the table format stays, the data stays; what changes is the compute layer.

That does not mean every technology change is free. Query syntax, performance tuning, connectors and operational processes may all need adapting. But the architecture is designed to make the change possible.

Cloud vendor lock-in

Lock-in does not form only in data technologies; it forms at the infrastructure layer too. Where a data platform's storage, orchestration, processing and monitoring layers are built only on one cloud provider's proprietary services, moving to another environment gets harder.

The Kubernetes-based Kauzas approach aims to let platform services run on different infrastructure. The same platform approach holds in:

• On-premise
• Private cloud
• Public cloud
• Hybrid cloud

The aim here is not to avoid the cloud.

The aim is to keep the difference between using the cloud and depending on it.

What Kubernetes delivers here, and what it does not →

On-premise and the open lakehouse

One of the open lakehouse's advantages is that it needs no particular public cloud infrastructure. With S3-compatible object storage, open table formats and compute technologies running on Kubernetes, a lakehouse architecture can be built in the organization's own data centre.

Trino's lakehouse connectors, for instance, work with a range of storage systems, S3-compatible object storage among them. The same underlying data architecture therefore holds across different environments, according to the organization's infrastructure strategy.

More on the on-premise AI platform →

The open lakehouse and AI

AI makes an open data architecture matter more, because AI technology changes quickly. The components in use today may be different ones in a few years:

• The embedding model
• The vector database
• The LLM
• The agent framework
• The ML engine

Enterprise data bound tightly to any one of these can make new AI technologies harder to adopt. In the open lakehouse approach the data layer is kept separate from the AI technologies.

Explore the Kauzas AI Platform →

The open lakehouse and enterprise RAG

The enterprise knowledge an RAG system needs may over time be used by different AI models and retrieval technologies.

Keeping that data in an open lakehouse layer means AI applications can be built without binding their data infrastructure to a single model or AI provider. Structured data is queried through the lakehouse while unstructured knowledge is reached by other retrieval methods.

More on enterprise RAG →

The open lakehouse and AI agents

AI agents can run queries against enterprise data and use different data services as tools.

An open data architecture supports architectures where the agent layer uses core enterprise data without moving it into some AI framework's proprietary data layer. Which data an agent may reach is then controlled by separate governance and authorization policy.

More on AI agent governance →

The open lakehouse against a tightly coupled platform

The difference between the two approaches fits in one table:

The last row belongs in the table too: an open architecture is paid for in operations.

Does an open architecture need more operations?

Usually, yes. Standing up and running a data platform made of several open technologies on your own brings real responsibilities:

• Deployment
• Upgrades
• Dependency management
• Security
• Monitoring
• Scaling
• Integration

This is where Kauzas comes in: keeping the flexibility of open technologies while making them manageable within a single platform experience.

The choice, in other words, need not be between open source and an enterprise platform.

Open architecture with an enterprise platform experience.

The Kauzas open data lakehouse

The Kauzas Data Platform offers an enterprise lakehouse approach built on open, replaceable technology layers. Depending on the use case, these layers work together within the architecture:

• Object storage
• Apache Iceberg
• Catalog
• Trino
• Apache Spark
• Orchestration
• Governance
• Observability

The platform runs on Kubernetes and holds across different infrastructure environments, so organizations can keep the flexibility of an open architecture while managing the data platform's operational complexity through Kauzas.

The lakehouse architecture in full →

Frequently asked questions

What is an open data lakehouse?

An open data lakehouse is a lakehouse architecture in which data is held in open file and table formats and different query, processing and AI technologies work against the same data layer.

What is vendor lock-in?

Vendor lock-in is what happens when changing the technology or platform in use becomes technically, operationally or economically hard.

Does Apache Iceberg prevent vendor lock-in?

Iceberg is an open table format and can be used by different compute engines, which helps separate the table and compute layers. Whether a system is free of lock-in altogether, though, does not rest on the table format alone.

Can an open lakehouse run on-premise?

Yes. With suitable object storage, catalog and compute technologies, an open lakehouse can run on the organization's own infrastructure.

Are there disadvantages to an open architecture?

Several technology components can add operational complexity in deployment, monitoring, upgrades and integration. Enterprise platform layers help manage that complexity.

Is Kauzas a proprietary data lakehouse?

The Kauzas approach is to govern different technology layers working on open data and table formats within one platform, rather than moving data into a proprietary Kauzas format.