Saurav Misra

Databricks announced Databricks Free Edition !! What is Databricks Free Edition? Databricks Free Edition is a no-cost version of Databricks designed for students, educators, hobbyists, and anyone interested in learning or experimenting with data and AI. Free Edition gives you an easy-to-use Databricks workspace where you can explore datasets, build and share projects, and work with AI and machine learning tools. It’s ideal for learning, prototyping, and collaborative exploration. https://lnkd.in/epsAc6rP #databricks

13

1 Comment

#DailyDataDose ☕💁‍♀️ – Day 9 : Python UDF vs Scala UDF in Spark – Same Logic, Different Speed In Spark, UDFs (User-Defined Functions) let you apply custom logic when built-in functions aren’t enough. But here’s something many overlook: The language you write your UDF in has a huge impact on performance. 🐍 #Python_UDF ✔ Easy to write ❌ Runs outside the Spark engine (via Py4J) ❌ Slower due to JVM ↔ Python communication ⚠️ Can be a bottleneck at scale 🟦 #Scala_UDF ✔ Runs natively in the JVM ✔ Close to Spark’s execution engine ✔ Much faster and better optimized ❌ Less accessible if you don’t code in Scala 💡 Real Example: Mandelbrot Benchmark We implemented the exact same function in both Python and Scala a CPU-heavy logic to simulate real-world processing. ⏱ Results on 100K rows: Python UDF: 7.07 sec Scala UDF: 1.24 sec 👉 That’s nearly 6x faster using Scala! 🔍 Why such a difference? Every time Spark calls a Python UDF, it sends the data through a bridge to a separate Python process. That means more serialization, more latency, and more overhead. Scala UDFs run inside the engine no bridge, no extra cost. Same logic, different results. Choosing the right language can cut execution time by 80%. #DailyDataDose #SparkTips #DataEngineering #PythonUDF #ScalaUDF #BigData #Optimisation #ApacheSpark

22

4 Comments

You must have used computes in Databricks but might not be aware that different kinds of computes that are available in databricks. Let me bring 𝐚𝐥𝐥 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐤𝐢𝐧𝐝𝐬 𝐨𝐟 𝐜𝐨𝐦𝐩𝐮𝐭𝐞𝐬 available in databricks in this single post/page so that this post can become a single source of your interview preparation for computes. The purpose,uses and benefit of all different kinds of compute is very different.Let me list down all those below in point wise manner. 1)𝐒𝐞𝐫𝐯𝐞𝐫𝐥𝐞𝐬𝐬 𝐜𝐨𝐦𝐩𝐮𝐭𝐞 𝐟𝐨𝐫 𝐧𝐨𝐭𝐞𝐛𝐨𝐨𝐤𝐬: On-demand, scalable compute used to execute SQL and Python code in notebooks. 2)��𝐞𝐫𝐯𝐞𝐫𝐥𝐞𝐬𝐬 𝐜𝐨𝐦𝐩𝐮𝐭𝐞 𝐟𝐨𝐫 𝐣𝐨𝐛𝐬: On-demand, scalable compute used to run your Lakeflow Jobs without configuring and deploying infrastructure. 3)𝐀𝐥𝐥-𝐩𝐮𝐫𝐩𝐨𝐬𝐞 𝐜𝐨𝐦𝐩𝐮𝐭𝐞: Provisioned compute used to analyze data in notebooks. You can create, terminate, and restart this compute using the UI, CLI, or REST API. 4)𝐉𝐨𝐛𝐬 𝐜𝐨𝐦𝐩𝐮𝐭𝐞: Provisioned compute used to run automated jobs. The Azure Databricks job scheduler automatically creates a job compute whenever a job is configured to run on new compute. The compute terminates when the job is complete. You cannot restart a job compute. 5)𝐈𝐧𝐬𝐭𝐚𝐧𝐜𝐞 𝐩𝐨𝐨𝐥𝐬: Compute with idle, ready-to-use instances, used to reduce start and autoscaling times. You can create this compute using the UI, CLI, or REST API. 6)𝐒𝐞𝐫𝐯𝐞𝐫𝐥𝐞𝐬𝐬 𝐒𝐐𝐋 𝐰𝐚𝐫𝐞𝐡𝐨𝐮𝐬𝐞𝐬: On-demand elastic compute used to run SQL commands on data objects in the SQL editor or interactive notebooks. You can create SQL warehouses using the UI, CLI, or REST API. 7)𝐂𝐥𝐚𝐬𝐬𝐢𝐜 𝐒𝐐𝐋 𝐰𝐚𝐫𝐞𝐡𝐨𝐮𝐬𝐞𝐬: Used to run SQL commands on data objects in the SQL editor or interactive notebooks. You can create SQL warehouses using the UI, CLI, or REST API. These compute types are used in day-to-day activities in data engineering. If you are a Data Engineer and using Databricks,do comment which compute type you are using? Do follow Arun Kumar and ForumDE for high quality & to the point AI and Data Engineeiring contents for cracking high paying job intervews.

16

1 Comment

🚀 𝗦𝘂𝗽𝗲𝗿𝗰𝗵𝗮𝗿𝗴𝗲 𝘆𝗼𝘂𝗿 𝗔𝗽𝗮𝗰𝗵𝗲 𝗦𝗽𝗮𝗿𝗸 𝗱𝗮𝘁𝗮 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝘄𝗶𝘁𝗵 #𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗢𝘃𝗲𝗿𝘄𝗿𝗶𝘁𝗲𝗠𝗼𝗱𝗲! Enable Dynamic Partition Overwrite in Spark. This powerful feature, introduced in Spark 2.3, is a game-changer for dynamic partition overwrites.It offers two modes: * 𝗦𝗧𝗔𝗧𝗜𝗖: The default mode, which overwrites the entire partition. * 𝗗𝗬𝗡𝗔𝗠𝗜𝗖: Overwrites only the partitions included in your new data, saving significant time and resources. Key takeaways for best practices:  • Careful selection of partition columns.   • Utilizing bucketing for optimized data access.   • Maintaining schema consistency.   • Using efficient file formats like Parquet or ORC. By implementing #PartitionOverwriteMode, you can make your data pipelines faster, more reliable, and avoid unnecessary data rewrites. In the example below 👇, Spark will only update the partitions for year=2023 and month=08 and month=09, leaving other partitions untouched. This way, the partitions that don’t need updating are preserved, saving time and resources. #ApacheSpark #DataEngineering #BigData #SparkTips #DataManagement

141

4 Comments

Learning #40: 🚀 Understanding PySpark Task Execution: A Simple Breakdown Ever wondered how PySpark handles your data in Databricks? Let’s break it down with a practical case: 📂 Input File: 1.2 GB CSV ⚙️ Partitioning Logic: Spark splits files based on spark.sql.files.maxPartitionBytes (default: 128 MB) 📊 Estimated Partitions: 1.2 GB / 128 MB = ~10 partitions 🧠 Tasks Generated: 10 (1 per partition) 💻 Cluster Setup: 3 Executors 2 Cores per Executor → 6 concurrent tasks (1 per core) 🧪 What happens during execution? Round 1: 6 tasks run in parallel Round 2: Remaining 4 tasks run once cores become available 💡 Core Principle: 🔁 1 Core = 1 Concurrent Task So even with 10 tasks, only 6 run at once — the rest queue up. 🎯 Takeaway: Your performance doesn’t depend on executors alone. Understanding how Spark partitions data, how many cores are available, and how tasks get scheduled is key to optimizing your workloads. Whether you're working with S3, ADLS, or Delta Lake — these principles apply across modern data lake architectures. #bigdata #dataengineering #apachespark #data

17

Which one should I learn ADF, Synapse, or Databricks? This is a question I hear every week from IT professionals moving into data engineering. Each of these tools has its strengths. But the right choice depends on your background and where you want to go. Here’s a simple way to think about it 👇 🔹 Azure Data Factory (ADF) Best for: ETL developers, BI engineers Why use it: GUI-based pipelines, 100+ connectors, low-code friendly Watch out: Not built for real-time or advanced transformations 🔹 Azure Synapse Best for: BI teams, dashboard/reporting needs Why use it: Serverless SQL, easy ad-hoc analysis, great for curated data Watch out: Not ideal for streaming or machine learning pipelines 🔹 Databricks Best for: Cloud data engineers, ML workflows Why use it: Spark-native, supports real-time, strong with notebooks & Delta Lake Watch out: Higher learning curve, needs solid coding skills (Python, SQL, Spark) 👉 If you’re coming from a traditional dev or BI background, ADF or Synapse might feel familiar. But if you’re aiming to build modern, scalable pipelines that power AI and real-time apps Databricks is where the industry is headed. Want help deciding where to focus based on your career goals? Drop a comment or DM — happy to share what’s working for 100s of my mentees making this exact switch. Follow Sateesh Pabbathi for more! #DataEngineering #Databricks #Azure #ADF #Synapse #CareerSwitch #ETL #CloudCareers #Mentorship #LevelUpWithSateesh

42

6 Comments

🚀 Snowflake Interview Questions for Senior Data Engineers – Part 2 In Part 1, we covered Architecture, Performance Optimization, and Data Modeling. Now, let’s dive into Security, Advanced Features, and Cost Optimization — areas where senior engineers are expected to think strategically. 🔹 4. Data Ingestion & Integration What are the different ways to load data into Snowflake (e.g., COPY, Snowpipe, external tables)? Can you explain how Snowpipe works and when it’s the right choice? How do external stages (S3, Azure Blob, GCS) integrate with Snowflake? What’s the role of file formats (Parquet, ORC, JSON, CSV) in Snowflake ingestion? How do you design a robust ELT/ETL pipeline into Snowflake? 🔹 5. Security & Governance How do Snowflake roles and grants work? What is the purpose of network policies in Snowflake? How do you enable row-level security (RLS) and column-level security (CLS)? Can you explain masking policies and their use cases? How do you implement data sharing securely across accounts or organizations? 🔹 6. Advanced Features How do streams and tasks enable near real-time pipelines in Snowflake? Can you explain the difference between replication and database failover in Snowflake? What is Time Travel in Snowflake, and how is it useful in recovery scenarios? How does zero-copy cloning work, and when would you use it? How would you use Snowflake’s external functions with AWS Lambda or Azure Functions? 🔹 7. Monitoring & Cost Management How do you monitor warehouse usage and query performance in Snowflake? What techniques do you use to optimize Snowflake cost without impacting performance? How do you analyze query history and warehouse utilization to detect inefficiencies? What are the best practices for managing multi-cluster warehouses? How do you track storage costs, including Time Travel and Fail-safe data? 👉 Thinking of making Part 3 all about real-world scenarios in Snowflake — like, “Your costs just doubled, how do you figure out why?” Should I go ahead and share those? Follow Avinash S. for more. #Snowflake #DataEngineering #CloudData #InterviewPrep #SeniorDataEngineer Would you like me to also create a Part 3 post that includes Scenario-Based Questions (e.g., “Your Snowflake costs doubled last month, how would you investigate?”) for a more practical angle?

65

3 Comments

Firebolt v4.29 is here 🚀 1. Iceberg Table Writes (Public Preview) – Write directly to file-based Apache Iceberg tables. A major step forward in open data lakehouse architecture. Now in public preview – we want your feedback. 2. Cross-Database Queries – Query and join data across multiple databases within your account using simple three-part identifiers. Complex analysis just got a lot simpler. 3. AI-Powered Development – Meet the Firebolt Agent in your workspace. Chat mode, in-editor generation, error fixing, and code improvements – all built in. 4. 10x Faster Top-K Queries – Late materialization optimization now extends to top-k queries, delivering up to 10x performance gains on eligible workloads. 5. Enhanced Compression – Chain multiple compression codecs (Delta + LZ4, ZSTD with configurable levels) to optimize storage and query performance. 6. Dynamic Primary Indexes – Add, change, or drop primary indexes on existing tables with ALTER TABLE operations – no table recreation required. Plus: Metabase Cloud integration, configurable query cache sizes, and fully scale-out outer joins on multi-node engines. Ready to experience the fastest cloud data warehouse? Learn more about 4.29 → https://okt.to/D3hP0M #DataWarehouse #CloudData #Analytics #Firebolt #ApacheIceberg

13

🔍 Delta vs Hudi vs Iceberg — Choosing the Right Table Format: Modern data lakes are no longer just Parquet files in object storage. Table formats like Delta Lake, Apache Hudi and Apache Iceberg bring ACID guarantees, schema evolution, and time travel but they are not interchangeable. Here’s a high-level breakdown I have found most useful when comparing them in real-world data engineering systems: ✅ 1. Delta Lake — Optimized Lakehouse for Spark & Databricks Delta Lake is tightly integrated with Databricks and Spark, making it a strong default choice for lakehouse architectures. Key strengths: → ACID transactions with a simple transaction log → Excellent support for MERGE, UPDATE, DELETE → Deep integration with Spark & Unity Catalog Best fit: 👉 Spark-centric analytics and ETL workloads on Databricks ✅ 2. Apache Iceberg — Built for Multi-Engine Analytics Iceberg is designed from day one to support multiple compute engines reading and writing the same table safely. Key strengths: → Snapshot-based metadata & manifest pruning → Strong multi-engine read/write (Spark, Flink, Trino, Snowflake, etc.) → Hidden partitioning for simpler data modeling Best fit: 👉 Organizations with Databricks + Trino/Flink/Snowflake ecosystems ✅ 3. Apache Hudi — Streaming & CDC at Scale Hudi shines when data is constantly changing and arriving out of order. Key strengths: → Efficient upserts and deletes → Designed for CDC and streaming ingestion → Copy-on-Write (COW) and Merge-on-Read (MOR) storage modes Best fit: 👉 High-frequency updates, CDC pipelines, near-real-time data lakes 💡 Key takeaway All three formats support ACID, schema evolution, and time travel. The real difference is workload optimization — not SQL features. Spark-first lakehouse → Delta Multi-engine analytics → Iceberg Streaming & CDC-heavy pipelines → Hudi 📌 Final thought: Choosing the right table format is less about what SQL you can run and more about how your data is written, queried, and shared across engines. How is your team deciding between Delta, Iceberg, and Hudi today? #Databricks #DataEngineering #Apache #Hudi #Iceberg #Delta #C2C #c2h #Hiring #Directclient #USITJobs

29

𝗦𝗻𝗼𝘄𝗳𝗹𝗮𝗸𝗲 𝗠𝗮𝘀𝘁𝗲𝗿𝘆 - 𝗗𝗮𝘆 𝟳 Sharing 𝗗𝗮𝘆 𝟳 of the 𝗦𝗻𝗼𝘄𝗳𝗹𝗮𝗸𝗲 learning series. Today’s document focuses on 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗿𝗲𝗮𝗱𝘆 𝗮𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻 𝗰𝗼𝗻𝗰𝗲𝗽𝘁𝘀 𝗶𝗻 𝗦𝗻𝗼𝘄𝗳𝗹𝗮𝗸𝗲 - features that help build reliable and scalable data pipelines. What you’ll find in today’s document: • Streams for Change Data Capture (CDC) • 𝗧𝗮𝘀𝗸𝘀 𝗳𝗼𝗿 𝘀𝗰𝗵𝗲𝗱𝘂𝗹𝗶𝗻𝗴 automated SQL workflows • 𝗧𝗮𝘀𝗸 𝗱𝗲𝗽𝗲𝗻𝗱𝗲𝗻𝗰𝗶𝗲𝘀 (𝗗𝗔𝗚𝘀) for pipeline orchestration • 𝗦𝘁𝗼𝗿𝗲𝗱 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗲𝘀 (𝗝𝗮𝘃𝗮𝗦𝗰𝗿𝗶𝗽𝘁) for complex logic • 𝗨𝘀𝗲𝗿-𝗗𝗲𝗳𝗶𝗻𝗲𝗱 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀 (𝗨𝗗𝗙𝘀) for reusable logic • 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗘𝗟𝗧 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗶𝗻𝘀𝗶𝗱𝗲 𝗦𝗻𝗼𝘄𝗳𝗹𝗮𝗸𝗲 These concepts are widely used in real production environments to 𝗮𝘂𝘁𝗼𝗺𝗮𝘁𝗲 𝗱𝗮𝘁𝗮 𝗽𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀 𝗮𝗻𝗱 𝗺𝗮𝗶𝗻𝘁𝗮𝗶𝗻 𝗱𝗮𝘁𝗮 𝗳𝗿𝗲𝘀𝗵𝗻𝗲𝘀𝘀 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁𝗹𝘆. Snowflake_Day7_Streams_Tasks_Pr… If you’re learning Snowflake from a 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗽𝗲𝗿𝘀𝗽𝗲𝗰𝘁𝗶𝘃𝗲, these features are important to understand after mastering core querying. Tomorrow’s topic: 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 & 𝗖𝗼𝘀𝘁 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗶𝗻 𝗦𝗻𝗼𝘄𝗳𝗹𝗮𝗸𝗲

47

2 Comments

You will be 100x better in Snowflake if you master the following: 𝐃𝐚𝐭𝐚 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 & 𝐂𝐥𝐨𝐮𝐝 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧 ↳ Snowflake virtual warehouse sizing/auto-scaling ↳ Cloud storage integration (S3, Blob, GCS) with stages ↳ Data sharing/secure data marketplace implementation ↳ Role-based access control (RBAC) & data masking 𝐃𝐚𝐭𝐚 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 𝐄𝐱𝐜𝐞𝐥𝐥𝐞𝐧𝐜𝐞 ↳ Hybrid tables for row-level operations ↳ Clustered column optimization for query pruning ↳ Time Travel & zero-copy cloning implementations ↳ Dynamic data masking & tag-based governance 𝐃𝐚𝐭𝐚 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐌𝐚𝐬𝐭𝐞𝐫𝐲 ↳ Snowpipe for real-time ingestion ↳ Streams & Tasks for CDC pipelines ↳ Snowpark ML for in-warehouse model training ↳ Cortex ML functions for anomaly detection 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐚𝐭𝐬𝐭𝐢𝐨𝐧 ↳ Query profiling with EXPLAIN plans ↳ Materialized view strategy design ↳ Micro-partition optimization techniques ↳ Result cache utilization & warehouse scaling 𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧-𝐆𝐫𝐚𝐝𝐞 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬 ↳ CI/CD for schema changes (GitHub Actions) ↳ Cost monitoring with resource monitors ↳ Data lineage tracking & version control ↳ Multi-cluster warehouse management Attached is an excellent doc by Shwetank Singh If you've read so far, do LIKE the post 👍 𝐏.𝐒: Learn no-nonsense AWS Data Engineering with 409+ others: https://aws.sachin.cloud & 𝐖𝐡𝐚𝐭𝐬𝐚𝐩𝐩 group for notification : https://w.sachin.cloud (phone privacy ensured, you won't see others )

242

25 Comments

Some good questions around data modeling and data warehousing:- 1. What’s the difference between 𝐎𝐋𝐓𝐏 and 𝐎𝐋𝐀𝐏 systems? 2. How do you design a 𝐬𝐜𝐚𝐥𝐚𝐛𝐥𝐞 data warehouse architecture? 3. What’s the difference between conceptual, logical, and physical data models? 4. 𝐒𝐭𝐚𝐫 schema vs 𝐒𝐧𝐨𝐰𝐟𝐥𝐚𝐤𝐞 schema — when to choose which? 5. What are 𝐟𝐚𝐜𝐭 and 𝐝𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧 tables? Give real-world examples. 6. How do you decide the grain of a fact table? 7. What types of fact tables exist — 𝘵𝘳𝘢𝘯𝘴𝘢𝘤𝘵𝘪𝘰𝘯𝘢𝘭, 𝘱𝘦𝘳𝘪𝘰𝘥𝘪𝘤 𝘴𝘯𝘢𝘱𝘴𝘩𝘰𝘵, 𝘰𝘳 𝘢𝘤𝘤𝘶𝘮𝘶𝘭𝘢𝘵𝘪𝘯𝘨 𝘴𝘯𝘢𝘱𝘴𝘩𝘰𝘵? 8. How do you handle slowly changing dimensions 𝐒𝐂𝐃 Types 1, 2, and 3? 9. What is data 𝐝𝐞𝐧𝐨𝐫𝐦𝐚𝐥𝐢𝐬𝐚𝐭𝐢𝐨𝐧, and when should you apply it? 10. How do 𝐬𝐮𝐫𝐫𝐨𝐠𝐚𝐭𝐞 keys differ from 𝐧𝐚𝐭𝐮𝐫𝐚𝐥 keys in data modeling? 11. What are junk, role-playing, and degenerate dimensions? 12. How would you design a schema for tracking user activity or clickstream data? 13. What’s the role of 𝘴𝘵𝘢𝘨𝘪𝘯𝘨, 𝘖𝘋𝘚, 𝘢𝘯𝘥 𝘱𝘳𝘦𝘴𝘦𝘯𝘵𝘢𝘵𝘪𝘰𝘯 layers in a warehouse? 14. How do you ensure 𝐝𝐚𝐭𝐚 𝐪𝐮𝐚𝐥𝐢𝐭𝐲 and consistency across multiple ETL layers? 15. What are the key 𝐩𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐭𝐮𝐧𝐢𝐧𝐠 strategies for large-scale dimensional models? Happy Learning 😊 #DataEngineering #DataModeling #DataWarehouse #ETL #BigData #Analytics #SeniorDataEngineer #TechLeadership #DataArchitecture #CareerGrowth #Interviews #Data #Datascience

6