From the course: Rust for Data Engineering

Unlock this course with a free trial

Join today to access over 25,300 courses taught by industry experts.

Core components of data engineering pipelines

Core components of data engineering pipelines - Rust Tutorial

From the course: Rust for Data Engineering

Start my 1-month free trial Buy for my team

Core components of data engineering pipelines

“

- [Instructor] Here we have a data engineering pipelines diagram where I've been able to whittle things down to really three different components of a data engineering pipeline. First up we have data ingestion, second up, data processing, third, data storage. So these are really the uniform aspects of a data engineering pipeline that are always present. The orange items here, resilience, scalability, monitoring, these are components that are really persistent in that they're going to be a part of any data engineering pipeline that goes into production. So let's go ahead and talk through each of these components. So first up, in terms of the ingestion, this is the most important part of a data pipeline. Really it's, can I get anything to work? And the data can come from multiple sources. It could be a database, a data lake, or even real-time data streams. And it's going to be able to process that data from a…

Contents