From the course: Data Engineering Project: Build Streaming Ingestion Pipelines for Snowflake with AWS
Join today to access over 24,800 courses taught by industry experts.
Introduction to streaming ingestion pipelines
From the course: Data Engineering Project: Build Streaming Ingestion Pipelines for Snowflake with AWS
Introduction to streaming ingestion pipelines
- [Instructor] Data pipelines are the core of a data engineer's everyday workflows. While batch pipelines have been a staple workflow in the data engineering space for decades, streaming data pipelines are gaining popularity with the International Data Corporation reporting that by 2025, 30% of all data generated will be near real-time. In this chapter, we'll dive deeper into streaming data pipelines. Before we get into streaming data pipelines, let's start with data pipelines, more broadly speaking. Data pipelines are the coordination of infrastructure, tools, and processes that move data from a source, such as a transaction service, or enterprise application, through the ELT process, extract, load, and transform, and into a destination. This destination could be a warehouse or lake house, such as snowflake. Even further downstream, that data could be consumed by visualization tool, or ML model, or reverse ETL back…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.