From the course: Google Cloud Professional Data Engineer Cert Prep (2025)

Unlock this course with a free trial

Join today to access over 25,300 courses taught by industry experts.

Data engineering pipelines

Data engineering pipelines - Google Cloud Platform Tutorial

From the course: Google Cloud Professional Data Engineer Cert Prep (2025)

Start my 1-month free trial Buy for my team

Data engineering pipelines

“

- [Instructor] Here we have a data engineering pipelines diagram where I've been able to whittle things down to really, three different components of a data engineering pipeline. First up we have data ingestion, second up, data processing, third, data storage. So these are really the uniform aspects of a data engineering pipeline that are always present. The orange items here, resilience, scalability, monitoring, these are components that are really persistent and that they're going to be a part of any data engineering pipeline that goes into production. So let's go ahead and talk through each of these components. So first up, in terms of the ingestion, this is the most important part of a data pipeline. Really it's, can I get anything to work? And the data can come from multiple sources. It could be a database, a data lake, or even real time data streams. And it's going to be able to process that data from a collection…

Contents