From the course: Data Engineering Project: Build Streaming Ingestion Pipelines for Snowflake with AWS
Join today to access over 24,800 courses taught by industry experts.
Streaming data pipelines: A deeper dive
From the course: Data Engineering Project: Build Streaming Ingestion Pipelines for Snowflake with AWS
Streaming data pipelines: A deeper dive
- [Instructor] In our previous video, we walked through an overview of streaming data pipelines. In this video, we'll dive deeper into the anatomy of a streaming architecture, considerations and practical use cases. Let's briefly look at the core anatomy of a streaming pipeline. Streaming pipelines generally consist of a provider or set of providers. This could be your products backend, your order service, or IOT sensors. Your streaming platform could pull from this client or the client may publish to this via an application programming interface or API. A topic is a group of events that are related to each other to which data is published by producers. Technically speaking, a topic is a log of events that are appended only and immutable. You can think of a topic as a table name in your destination database. Of course, you have the actual event streaming service. This can be distributed for better fault tolerance or…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.