From the course: End-to-End Data Engineering Project

Project architecture overview

From the course: End-to-End Data Engineering Project

Start my 1-month free trial Buy for my team

Project architecture overview

“

- Welcome to the team, fellow data engineer. You are now part of Big Star Collectibles, a vibrant e-commerce company. Ahead of you lies a rewarding task that is critical for the company's growth. Our marketing team is eager to drive our next campaign, but they are in the dark. Why? They need data, specific actionable data. And there's no data engineering team in the company. The data is sitting in a Postgres backend database. It is raw and not very useful for the marketing team. Not to mention, it is not optimized for analytics. The marketing team needs a holistic view, so we will combine and transform some of the tables in the database and make sure they are always up to date in our data warehouse. The aim is to provide a more comprehensive view of the customers' buying patterns. So how do we do this? We are going to set up a data warehouse and build a pipeline from scratch using some of the best tools in the modern data stack. Here is the plan. First, we will use Airbyte to extract data from the Postgres database and load it into BigQuery, our new data warehouse. Once the data is in BigQuery, we will clean it up and shape it. dbt will be our transformation tool, helping us turn the raw, scattered data into a neat table that provides insights about our customers. It'll also allow us to test and document our data models. And finally, we'll use Dagster and its user-friendly interface, Dagit, to ensure everything runs smoothly and in the right sequence. By the end of this course, you will be knowledgeable about these modern tools and you will also have built a production-grade data stack from scratch. Ready to take on this challenge?

Contents