Christopher Bergh’s Post

Test Coverage is enabled by Open Source Tools like DataKitchen’s DataOps Data Quality TestGen, and Shift Down With DataKitchen’s Open Source DataOps Observability https://hubs.ly/Q03XKb3k0 #dataquality #opensource #dataobservability #dataops #dataengineering

Test Coverage is enabled by Open Source Tools like DataKitchen’s DataOps Data Quality TestGen, and Shift Down With DataKitchen’s Open Source DataOps Observability https://hubs.ly/Q03Z6_Rt0 #dataquality #opensource #dataobservability #dataops #dataengineering

Test Coverage is enabled by Open Source Tools like DataKitchen’s DataOps Data Quality TestGen, and Shift Down With DataKitchen’s Open Source DataOps Observability https://hubs.ly/Q03XKlv40 #dataquality #opensource #dataobservability #dataops #dataengineering

Test Coverage is enabled by Open Source Tools like DataKitchen’s DataOps Data Quality TestGen, and Shift Down With DataKitchen’s Open Source DataOps Observability https://hubs.ly/Q03XKjqd0 #dataquality #opensource #dataobservability #dataops #dataengineering

Test Coverage is enabled by Open Source Tools like DataKitchen’s DataOps Data Quality TestGen, and Shift Down With DataKitchen’s Open Source DataOps Observability https://hubs.ly/Q03Z6-WK0 #dataquality #opensource #dataobservability #dataops #dataengineering

Test Coverage is enabled by Open Source Tools like DataKitchen’s DataOps Data Quality TestGen, and Shift Down With DataKitchen’s Open Source DataOps Observability https://hubs.ly/Q03Z6_7S0 #dataquality #opensource #dataobservability #dataops #dataengineering

Day 15 of IDC 21 days of SQL challenge sponsored by #DPDzero Day 15 challenge: "Create a comprehensive service analysis report  for week 20 showing: service name, total patients admitted  that week, total patients refused, average patient satisfaction,  count of staff assigned to service,  and count of staff present that week. Order by patients admitted descending." 💡 What I learned today ⬇️ - Joining of Multiple tables - Importance of DISTINCT in counting the rows. - Importance of step by step joining of each tables. Query ⬇️ SELECT sw.service AS service_name,   sw.patients_admitted AS total_patients_admitted,   sw.patients_refused AS total_patients_refused,   COUNT(DISTINCT s.staff_id) AS staff_assigned, ROUND(AVG(p.satisfaction),0) AS avg_patient_satisfaction,   COUNT(DISTINCT CASE WHEN ss.present='Yes' THEN ss.staff_id END) AS staff_present_week20 FROM services_weekly sw LEFT JOIN patients p ON sw.service=p.service AND week(p.arrival_date)=20 LEFT JOIN staff s ON s.service=sw.service LEFT JOIN staff_schedule ss ON s.staff_id=ss.staff_id AND ss.week=20 WHERE sw.week=20 GROUP BY  sw.service,   sw.patients_admitted,   sw.patients_refused ORDER BY total_patients_admitted DESC ⏩ My takeaway: Thinking and writing query by considering all the table to be joined is overwhelming. So the technical mindset much needed to solve complex problems. Thanks for giving this opportunity Indian Data Club DPDzero Codebasics

Are hidden data errors slowing your decisions? Explore proven data validation rules and services that ensure accuracy using the right methods, tools, and transformations: #Data #DataValidation #Verification #DataMethods #DataTools #DataTransformation #DataQuality #Eminenture #dataverification

Small files silently killing your data platform's performance, do you know that? The Problem: Your batch jobs are crawling because they spend up to 70% of their execution time simply *listing* thousands of tiny files instead of actually processing the data. The Solution: You don't need expensive, "sexy" tools to fix this; you just need to implement the Compactor pattern to merge those small files into larger, I/O-efficient ones. Don't let metadata overhead bottleneck your pipeline. Compact your data. #DataEngineering #ApacheSpark #DeltaLake #BigData #Optimization #DesignPatterns