From the course: Databricks Certified Data Engineer Associate Cert Prep
Join today to access over 25,300 courses taught by industry experts.
Task dependencies
From the course: Databricks Certified Data Engineer Associate Cert Prep
Task dependencies
- [Instructor] Here we have a workflow that involves multiple tasks, and that's one of the things you can do with the jobs framework. We have a songs workflow, a prepare songs data, and analyze songs data task. Each of these tasks, if I toggle through, has different types of dependencies. Notice at the very end here, that this one actually depends on the previous task. Likewise, if I go to this, it depends on the task before it. So this is part of building a task pipeline that involves multiple steps is that you have to wait until the previous step is complete, and when it's successfully completed, then it will proceed to the next step. This is where, also, you have to be careful of other task dependencies, other than just the task themselves. Some of the things that they could involve and involve dependent libraries, so for example, you may want a library loaded and it must be automatically attached to the cluster,…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.