From the course: End-to-End Data Engineering Project

What you should know

From the course: End-to-End Data Engineering Project

Start my 1-month free trial Buy for my team

What you should know

“

- Before we dive deep into the project, let's discuss a few things you should know. First, let's talk about knowledge base. To make the most of this course, you should be familiar with data engineering foundations, have a basic understanding of SQL and Python, and comfortable with basic command line operations and Git. About the tech stack, you will need Python 3, Docker and Docker Compose, which are bundled with Docker Desktop, and an IDE. I highly recommend Visual Studio Code. You can find links on how to install all of these tools in the Exercise File. We will build this data engineering project together from scratch. The main takeaway from this course is a GitHub Repo to showcase what you built. You will be able to understand and explain each and every section of it. To get you started, I provide an empty repo in the Main branch. At any point, you can consult the finished version of the project by checking out the Finished branch. And for those moments when I'm coding or entering lengthy commands, there's no need to pause a video and try to copy from the screen. The necessary code and commands are included in the Exercise Files as text. On the surface, this may come across as a very simple project, yet it is holistic, production-ready, and incorporates essentials, like documentation and testing, making it capable of handling massive data loads, if provided. It is meant to be a stepping stone for you to add more sources, more transformations, and make it your own. In the world of data engineering and especially the modern data stack, things change fast. If by the time you are building this project that things don't look the same, please don't get discouraged. The fundamentals will certainly remain the same. Whether you're an experienced data engineer looking to know new tools or a beginner looking to get some hands-on experience, this course is for you. Even if you're not a data engineer, you can still certainly expand your knowledge.

Contents