From the course: Implementing Data Engineering Solutions Using Microsoft Fabric (DP-700) Cert Prep by Microsoft Press
Join today to access over 25,300 courses taught by industry experts.
Configuring data workflow settings - Microsoft Fabric Tutorial
From the course: Implementing Data Engineering Solutions Using Microsoft Fabric (DP-700) Cert Prep by Microsoft Press
Configuring data workflow settings
Now with data workflow settings, this is specific to Apache Airflow. Now to find the Apache Airflow runtime settings, that's in the workspace settings, data factory tab. Now there are two types of pools that we can set for our runtime, a starter pool and a custom pool. we can create one or more custom pools and then select that custom pool in our runtime settings. Now, with starter pools, like Spark starter pools, they offer an instant compute startup time. It's automatically deprovisioned after the Airflow runtime has finished its job, so we don't have to switch off starter pools. Now, with custom pools, they offer us more flexibility. And what we can do is we can set either a large or small for our custom pools and enable auto scaling from between 2 to 20 nodes. Now, let's look at a table that shows us the differences between the starter pools and the custom pools. Now, as you can see, the size, we're set in our starter pool. It is large, and we can't configure that. Whereas the…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.