Observability for data engineering - Snowflake Tutorial

Years ago, I worked as a transmission protection engineer for one of the biggest public utilities in the United States. I was tasked with programming relays to make sure transmission lines of the highest voltages recovered independently when there was an electrical fault. I thought this was a huge responsibility at the time, and it was, but it paled in comparison to what I saw when I visited the headquarters. There was a huge room full of hundreds of people keeping an eye on all sorts of things on many different screens 24/7. There were groups keeping track of load on the system, groups keeping track of outages, weather, and much more. Being able to keep track of dimensions like these was crucial to making sure the electrical grid maintained strict uptime requirements for customers. There are many parallels in the data engineering world. Data engineers must frequently keep track of important data pipeline metrics to understand pipeline performance and to take action when something is out of line. This practice is known as Observability. Observability specifically means employing tools and practices to be able to actively and continuously monitor the flow of data through a data pipeline. When properly implemented, it can help data engineers diagnose issues so that they can take action and drop solutions when needed. At a high level, observability helps data engineers keep an eye on a pipeline's health and quality. By now, you know that pipelines are made up of many different parts. They're made up of ingestion logic, code for transformations, automation schedules, delivery methods, integrations with additional tools, and much more. This also means that there are many places where a pipeline can potentially fail. As a data engineer, you likely want to be able to take proactive action to ensure things don't fail rather than reactively take action when something does fail. This is a key reason why pipeline observability is core to building resilient pipelines. So what exactly are data engineers observing in their pipelines? Well, all sorts of things. For example, it's common to keep an eye on things like failures. This is a big focus and data engineers frequently track failures related to data quality, data loading, failed queries, connection failures, scheduling hiccups, and much more. Throughput: this refers to the rate at which data is being transferred through the pipeline. Has it dropped? If so, why? And what's being affected downstream? Latency: this is a bit related to throughput, focusing specifically on the speed of certain aspects. This could be keeping an eye on the rate at which data makes it into Snowflake or the latency related to a query or maybe the end-to-end latency of the pipeline. Is it successfully delivering its outputs at the required speed? And if not, is something getting in the way? Compute utilization, meaning, is the pipeline making the best use of the compute resource associated with it? Does it need to change? Cost: you might also keep an eye on the cost of a pipeline, specifically related to the cost to transfer data, any computationally complex queries, and general usage of compute resources. These are some examples but they generally highlight the core of what data engineers might track for a pipeline. So how exactly do data engineers implement observability into their pipelines to keep track of metrics like these? Join me in the next video to learn about some of the fundamental techniques for doing this.