Data Management

Data Lakehouse

A data lakehouse is a data platform that stores all data as open files in low-cost object storage while adding the tables, transactions, and governance of a data warehouse on top.

What is a data lakehouse?

A data lakehouse is a data architecture that keeps every kind of data in one low-cost storage layer and adds a table format on top so that the same files can be queried, updated, and governed like a database. It takes the cheap, flexible storage of a data lake and the reliable, structured tables of a data warehouse and puts them in a single system.

The piece that makes this possible is an open table format such as Delta Lake, Apache Iceberg, or Apache Hudi. The format sits over ordinary Parquet files and adds a transaction log, which gives the files the properties a database user expects: atomic writes, schema enforcement, the ability to read the table as it stood at an earlier point, and safe concurrent access by several tools. Because the files and the format are open, a SQL engine, a Python notebook, a business intelligence tool, and a machine learning job can all read the same table without copying it.

Data usually flows through the lakehouse in stages: raw extracts land unchanged, a cleaned and conformed version is built from them, and business-ready tables are built from that. The medallion architecture names these stages bronze, silver, and gold.

How a data lakehouse works

Ingestion jobs copy data from source systems into the raw layer, either in nightly batches or as a continuous feed through change data capture. Transformation jobs, written in SQL or Python and run by a data orchestration tool, read the raw tables and write curated ones. A catalog records every table, its schema, its owner, and who may read it. Query engines attach to the catalog and read the files directly from storage, scaling compute up for a heavy month-end close and back down afterward. Storage and compute are billed separately, so keeping ten years of history costs little while nobody is querying it.

Why a data lakehouse matters

Before lakehouses, companies that wanted both analytics and machine learning ran a warehouse for the first and a lake for the second, copied data between them, and spent effort keeping the copies in agreement. The lakehouse removes the copy, so finance reports and model training read the same governed tables, and a forecast and the dashboard it is judged against start from identical numbers.

For generative AI the benefit is similar. Retrieval-augmented generation needs documents, the tables those documents refer to, and the permissions that govern both, and holding them in one platform with one access model makes permission-aware retrieval practical. An agent that answers a question about a customer’s open orders reads a gold table that finance already trusts, rather than a private extract that drifts out of date.

Data lakehouse compared with data warehouse and data lake

Capability Data lake Data warehouse Data lakehouse
Storage cost Low Higher, proprietary Low, open files
Transactions and schema enforcement No Yes Yes, through the table format
Unstructured data Yes Poorly Yes
Machine learning access Direct file reads Export or connector Direct reads of governed tables
Vendor lock-in Low High Low to moderate

Data lakehouse in practice

A manufacturer with three plants on three ERP systems could not produce a single margin report without weeks of spreadsheet work. The team extracted each ERP nightly into bronze tables, mapped the three charts of accounts and item masters onto one canonical schema in a silver layer, and produced margin by product family and plant in gold. Controllers now open one dashboard the morning after close. The same silver tables feed a demand forecasting model and a text-to-SQL assistant that plant managers use to ask about scrap rates, with row-level security limiting each manager to their own plant.

Kirk Tech Solutions typically builds the lakehouse before the AI layer in engagements like this one, because agents and models are only as reliable as the tables they read.

What to watch for

  • Choose the table format deliberately, since the engines and vendors available later depend on it, and conversion is disruptive.
  • Insist on a catalog with access control from the first table, because retrofitting permissions onto hundreds of tables is slow and error-prone.
  • Treat the transformation code as software, with version control, tests, and reviews, since the gold tables will drive financial decisions.
  • Check how the platform handles small, frequent writes, because table formats create many small files that slow queries until they are compacted.
  • Ask whether a vendor’s lakehouse stores data in open formats other tools can read, or in a proprietary layout with the word attached.

← Back to AI Glossary