From the course: Complete Guide to Data Lakes and Lakehouses

What is a data lakehouse?

From the course: Complete Guide to Data Lakes and Lakehouses

Start my 1-month free trial Buy for my team

What is a data lakehouse?

“

- Imagine having the huge storage capacity of a data lake combined with the structure and capabilities of a data warehouse. That's what a data lakehouse is all about. A data lakehouse is an innovative data architecture that combines the best elements of data lakes and data warehouses. This hybrid model is designed to capitalize on the data storage and flexibility of data lakes, while incorporating the management features and analytics capabilities of data warehouses. It represents a shift in how data is stored, processed, and used, offering a single cohesive environment for all types of data analysis. But why did we need the data lakehouse in the first place? Data teams normally rely on data warehouses for structured data analytics and data lakes for storing raw, unstructured data. While they're quite effective in the respective roles, and they may perfectly fit certain use cases and specific needs, both systems have their own limitations. Data warehouses sometimes struggle with scalability and cost effectiveness when dealing with large volumes of data. This is in part due to the types and relationship between storage and compute, which is difficult to independently scale in some platforms. Data lakes, on the other hand, offer scale but lack the structure and readily available tools necessary for analytics, often turning into what's known as data swamps. The data lakehouse architecture emerges as a solution to those limitations I just mentioned and unifies all data use cases in a single platform. Essentially, the data lakehouse enhances the traditional data lake by adding a transactional layer that makes it behave as a data warehouse. But what are the architectural components behind it to make that possible? Let's take a look. At its core, the data lakehouse is built on top of the same storage technologies as a data lake. Its foundation is a unified storage layer that can handle huge amounts of structured, semi-structured, and unstructured data efficiently and cost effectively. The transactional layer is a component that the lakehouse implements and that manages how data is written and read, ensuring consistency and integrity through ACID's transactions, something that data lake cannot offer. We will talk about that in the next video. This layer allows multiple users and applications to concurrently interact with the lakehouse without compromising data reliability. The transactional layer is possible thanks to technologies such as Delta Lake by Databricks, Apache Iceberg, Apache Hudi, also called Table Formats, which introduce ACID transactions and schema enforcement at scale. An important architectural component of data lakehouses is implementation of dynamic schema management, which enforces schema-on-write and at the same time supports schema-on-read operations. Schema-on-write means that the data is structured as it is ingested, which makes it automatically usable for analytics and business intelligence similar to data warehouses. Schema-on-read means that the lakehouse maintains flexibility for data scientists to apply different schemas for advanced analytics characteristic of data lakes. To be able to execute complex queries directly on diverse data sets, data lakehouses include advanced query engines. These engines can support both batch and real-time data processing. Because of the architecture, data lakehouses are able to support business intelligence, machine learning, and AI use cases, effectively creating a unified analytics platform. This means you can transition between different types of data analysis, all within the same architectural framework. There's no wonder why data lake houses are quickly gaining traction, as they aim to offer the best of both worlds.

Contents