From the course: Implementing Data Engineering Solutions Using Microsoft Fabric (DP-700) Cert Prep by Microsoft Press
Join today to access over 25,300 courses taught by industry experts.
Monitoring data transformation - Microsoft Fabric Tutorial
From the course: Implementing Data Engineering Solutions Using Microsoft Fabric (DP-700) Cert Prep by Microsoft Press
Monitoring data transformation
monitoring data transformation. Now, if we look at Spark notebooks, we can monitor them in a monitoring hub and we can drill into the detail of that Spark notebook. Now, when we use the monitoring hub, we can look at details like the run details, the run series and the Spark history. Now, the run series actually gives us a breakdown of the notebook. So we can see the operations that that notebook is performing. We can see that over time. So we've got information such as the runtime version, the cause, the memory, and all the operations, such as read and write and shuffles, and see that spark activity. So we can see runs over the submitted time, the average run duration, anomalies in runs. Maybe a run took longer than it should have done, an average over time. We can see that Spark configuration and any of the activity types on the executors. We can also look at the Spark application information for an execution. Now, if we go into the jobs tab, that's gonna give us a drillable…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.