From the course: End-to-End Data Engineering Project

Unlock this course with a free trial

Join today to access over 25,300 courses taught by industry experts.

Solution: Add a schedule to your data pipeline

Solution: Add a schedule to your data pipeline

From the course: End-to-End Data Engineering Project

Start my 1-month free trial Buy for my team

Solution: Add a schedule to your data pipeline

“

(upbeat music) - [Instructor] Let me walk you through the steps I followed to solve this challenge. In the top level in each file, I created a new job using the define_asset_job method. In Dagster, a job allows us to focus on a set of assets and materialize them using a single action. Here I use Dagster's AssetSelection to select all available assets. Then I created a schedule. A schedule in Dagster triggers a jobs run at specific intervals. The interval in this case is given by the cron_scheduler. And last, I updated the definitions to include the new job and schedule. Let me launch Dagit and show you how this looks there. In the Dagster UI, I will navigate to the overview and schedules tab. And here is the newly added schedule linked to the job. There you have it, a schedule run for our data pipeline.

Contents