Deepak Goyal

🚀 TIL: `databricks bundle generate` is your fastest path to bring existing Databricks resources under bundle (IaC) control — but there's a sharp edge you MUST know before you hit deploy. 📦 What it does: Instead of handwriting YAML from scratch, `bundle generate` reverse-engineers your existing workspace resource — job, pipeline, dashboard, or app — into a ready-to-use bundle configuration file. It also pulls down referenced notebooks and source files into your local `src/` folder. ```bash databricks bundle generate job --existing-job-id 6565621249 --bind databricks bundle generate pipeline --existing-pipeline-id 7668611149 --bind databricks bundle generate dashboard --existing-id abc123 --bind databricks bundle generate app --existing-app-name my-app --bind ``` ⚠️ The Sharp Edge — Hardcoded IDs: The generated YAML snapshots your resource exactly as it exists today. That means fields like `pipeline_id`, `job_id`, and `existing_cluster_id` are baked in as static values. This is fine for a single environment. But the moment you try to promote this bundle from dev → staging → prod, those hardcoded IDs will point to the wrong workspace resource — or worse, fail silently. 🔧 The Fix — Parameterize with Bundle Variables: Replace every hardcoded ID with a bundle variable so each target environment resolves to the right resource at deploy time. ```yaml # ❌ Generated (hardcoded) pipeline_task:  pipeline_id: 7668611149d5709ac9 # ✅ Parameterized (correct) variables:  pipeline_id:   description: Target pipeline ID   default: 7668611149d5709ac9  # dev default resources:  jobs:   my_job:    tasks:     - pipeline_task:       pipeline_id: ${var.pipeline_id} targets:  prod:   variables:    pipeline_id: <prod-pipeline-id> ``` 🔗 And don't forget --bind: Without `--bind`, your next `bundle deploy` creates a brand-new duplicate resource instead of updating the existing one. Always bind immediately after generating — or run `databricks bundle deployment bind` separately before deploying. 💡 The right mental model: Think of `bundle generate` as a starting point, not a finish line. It gives you the scaffold; it's your job to replace environment-specific literals with `${var.*}` references before committing to source control. 👉 Have you migrated existing jobs or pipelines into bundles? What patterns worked for you? 👉 Checkout this LinkedIn post for Complete DAB with GitHub setup for "Production" grade CI/CD pipeline (Link available in the LinkedIn post) for your project:- https://lnkd.in/d-xUNdV6 Also Tagging Hubert Dudek for all important information's regarding Databricks DAB. Credit to him 🙏 for interest in exploring DAB (with GitHub ci/cd). #Databricks #DataEngineering #DatabricksAssetBundles #InfrastructureAsCode #LakehouseArchitecture #DataPlatform

16

1 Comment

You will be 100x better in Data Engineering If you master the following: 𝐃𝐚𝐭𝐚 𝐈𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐒𝐭𝐨𝐫𝐚𝐠𝐞   ↳ Auto Loader for Real-Time and Incremental File Ingestion   ↳ Delta Lake for ACID Transactions and Time Travel   ↳ Unity Catalog for Centralized Access Control   ↳ Partner Connect for Seamless Source Integration  𝐃𝐚𝐭𝐚 𝐏𝐫𝐨𝐜𝐞𝐬𝐬𝐢𝐧𝐠 𝐚𝐧𝐝 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧   ↳ PySpark and Databricks Runtime Optimization   ↳ Delta Live Tables (DLT) for Declarative ETL   ↳ Structured Streaming for Event-Driven Architectures   ↳ Notebooks for Modular and Collaborative Transformations  𝐁𝐢𝐠 𝐃𝐚𝐭𝐚 𝐓𝐞𝐜𝐡𝐧𝐨𝐥𝐨𝐠𝐢𝐞𝐬   ↳ Photon Engine for Lightning-Fast Queries   ↳ MLflow for End-to-End Experiment Tracking   ↳ Apache Spark Tuning and Adaptive Query Execution  𝐃𝐚𝐭𝐚 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 𝐚𝐧𝐝 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞   ↳ Medallion Architecture (Bronze ➝ Silver ➝ Gold Layers)   ↳ Schema Evolution in Delta Lake   ↳ Data Lineage and Auditing with Unity Catalog   ↳ Modular Pipeline Design Patterns  𝐃𝐞𝐩𝐥𝐨𝐲𝐦𝐞𝐧𝐭 𝐚𝐧𝐝 𝐎𝐫𝐜𝐡𝐞𝐬𝐭𝐫𝐚𝐭𝐢𝐨𝐧   ↳ Databricks Workflows for Task Automation   ↳ Git Integration with Databricks Repos   ↳ CI/CD with GitHub Actions or Azure DevOps   ↳ Job and Cluster Configuration for Cost Efficiency  𝐎𝐛𝐬𝐞𝐫𝐯𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐚𝐧𝐝 𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧 𝐑𝐞𝐚𝐝𝐢𝐧𝐞𝐬𝐬   ↳ Monitoring Jobs and Pipelines via Workflows UI   ↳ Logging with MLflow and Custom Log Aggregation   ↳ Alerting and Retry Strategies in Production   ↳ Billing Dashboards and Usage Insights for Optimization Top 9 Questions for Databricks! 1️⃣ How does Auto Loader perform incremental and real-time file ingestion in Databricks? 2️⃣ What are ACID transactions in Delta Lake, and how does Time Travel work? 3️⃣ How does Unity Catalog manage centralized access control and data lineage? 4️⃣ What are the key differences between Delta Live Tables and traditional PySpark ETL pipelines? 5️⃣ How does Structured Streaming ensure exactly-once processing semantics? 6️⃣ What is the Photon Engine, and how does it improve query performance in Databricks? 7️⃣ How would you design and implement a Medallion Architecture (Bronze → Silver → Gold) pipeline? 8️⃣ What strategies would you use to handle schema evolution in Delta Lake? 9️⃣ How can Databricks Workflows and Git integration be used to enable CI/CD for data pipelines? If you've read so far, do LIKE the post 👍   𝐏.𝐒: 575 + IT Professionals have already learnt real world AWS Data Engineering with us. Now it’s your turn—start with the Free AWS Data Engineering Interview Kit we’ve created for you. Get access in this community: https://w.sachin.cloud

76

10 Comments

𝗛𝗮𝘃𝗲𝗹𝗹𝘀 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 Most candidates prepare theory. But interviews? They test real scenarios. I found these scenario-based questions asked for Sr. Data Engineers Let’s break down what actually matters 👇 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗦𝗰𝗲𝗻𝗮𝗿𝗶𝗼 Dataset: 200GB Problem: Job running slow 𝗪𝗵𝗮𝘁 𝘁𝗵𝗲𝘆 𝗲𝘅𝗽𝗲𝗰𝘁: Use Spark UI → identify bottlenecks Convert to Delta / Parquet Apply partitioning Filter early (reduce shuffle) Use broadcast joins Cache reused data Tune cluster config 👉 They are testing: Optimization thinking 𝗔𝗗𝗙 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲 𝗙𝗮𝗶𝗹𝘂𝗿𝗲 Scenario: Pipeline failed overnight 𝗘𝘅𝗽𝗲𝗰𝘁𝗲𝗱 𝗮𝗽𝗽𝗿𝗼𝗮𝗰𝗵: Identify failed activity Check logs & error message Validate source (file path, schema, connectivity) Re-run intelligently Add alerts (Azure Monitor) Do RCA + prevention 👉 They are testing: Production debugging mindset 𝗜𝗻𝗰𝗿𝗲𝗺𝗲𝗻𝘁𝗮𝗹 𝗟𝗼𝗮𝗱 𝗗𝗲𝘀𝗶𝗴𝗻 𝗞𝗲𝘆 𝘀𝘁𝗲𝗽𝘀: Identify incremental column (timestamp) Store watermark Read only new data Append / merge to target 👉 They are testing: Data pipeline design clarity 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗝𝗼𝗯 𝗦𝗹𝗼𝘄 𝗜𝘀𝘀𝘂𝗲 𝗥𝗲𝗮𝘀𝗼𝗻𝘀: Data skew Auto scaling issues Heavy shuffle Small files problem No caching 👉 They are testing: Root cause analysis 𝗥𝗲𝗮𝗹 𝗶𝗻𝘀𝗶𝗴𝗵𝘁: If you only prepare syntax… you will struggle. If you prepare scenarios + reasoning… you’ll stand out. 🔄 Share this post if it adds value to your interview prep. Comment below 👇 Who is preparing for Data Engineer interviews right now? 𝗙𝗼𝗿 1:1 𝗴𝘂𝗶𝗱𝗮𝗻𝗰𝗲:- https://lnkd.in/gxjZT5P3 🚀 𝗟𝗲𝘃𝗲𝗹 𝘂𝗽 𝘆𝗼𝘂𝗿 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗽𝗿𝗲𝗽: 🔹 𝗦𝗤𝗟 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗠𝗮𝘀𝘁𝗲𝗿𝘆 – 200+ 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 https://lnkd.in/dYJb42RH 🔹 𝗟𝗶𝗻𝗸𝗲𝗱𝗜𝗻 + 𝗡𝗮𝘂𝗸𝗿𝗶 𝗣𝗿𝗼𝗳𝗶𝗹𝗲 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗕𝘂𝗻𝗱𝗹𝗲 https://lnkd.in/gqYBrbtS 🔹 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗲 – 𝟵𝟲 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 https://lnkd.in/dwCEinvH 🔹 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 𝗣𝘆𝘁𝗵𝗼𝗻 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 – 100 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 https://lnkd.in/dwKVeFf7 👉 Follow Anuj Shrivastav for more Data Engineering interview content. #DataEngineering #PySpark #Databricks #SQL #ADF #BigData #DataEngineer #InterviewPrep

69

12 Comments

Databricks DLT vs Azure Data Factory (ADF): 🔍 When to Use Databricks Delta Live Tables (DLT) Advanced Transformations — For heavy logic (joins, aggregations, window functions). Batch + Streaming — Unified support via AutoLoader. Event-Driven — Ideal for real-time or triggered data flows. Databricks Ecosystem — Tight integration with ML and notebooks. Data Quality Rules — Built-in EXPECT() checks. 🔍 When to Use Azure Data Factory (ADF) Drag & Drop ETL — Best for non-coders and quick jobs. Data Movement — Connects 100+ sources (SQL, Blob, SAP). Basic Transforms — Mapping Data Flows for filters, joins. Hybrid Integration — Supports on-premise via IR. Orchestration — Control flow, triggers, and dependencies.

4

🚀 Partitioning vs Clustering in Data Engineering — Made Simple In large datasets, smart data organization can turn slow queries into lightning-fast results. Two common techniques: Partitioning and Clustering. 1️⃣ Partitioning – “Separate folders” We split a table into chunks based on a column value. Example: 📂 /sales/country=USA/... 📂 /sales/country=UK/... Query: SELECT * FROM sales WHERE country = 'USA'; ✅ Only scans the USA folder — skips all UK data. Best for: Columns with few unique values (e.g., Country, Year). 2️⃣ Clustering – “Organized shelves” We keep the data in the same file but group similar rows together. ✅ Without Clustering The data might be stored unsorted, like: file1.parquet → (2024-01-03, USA), (2024-01-01, USA), (2024-01-02, UK)... file2.parquet → (2024-01-02, USA), (2024-01-01, UK), (2024-01-03, UK)... ✅ With Clustering by order_date The data inside each file is grouped and sorted by order_date: file1.parquet → 2024-01-01, USA, 100 2024-01-01, USA, 150 2024-01-02, UK, 200 2024-01-02, USA, 180 file2.parquet → 2024-01-03, UK, 220 2024-01-03, USA, 300 ✅ Queries can skip irrelevant ranges inside the file. Best for: Columns with many unique values (e.g., Order Date, User ID). 3️⃣ Together = Maximum Efficiency In BigQuery or Snowflake, you can do: PARTITION BY DATE(order_date) CLUSTER BY country This way: Partitioning skips entire days. Clustering skips irrelevant countries inside each day. 💡 Rule of Thumb: Partition for broad pruning (skip whole files) Cluster for fine-grained pruning (skip parts of files) 🔍 How do you use partitioning and clustering in your data projects? #DataEngineering #BigQuery #Snowflake #Partitioning #Clustering #SQLPerformance #DataModeling

15

Databricks is one of the most in-demand technology in Data Engineering right now. Recently I launched an 8 weeks in-depth Databricks Expert Program Here is the high level curriculum What is Databricks & Why Databricks Databricks Free Edition vs Azure Databricks High Level Architecture of Databricks Different Cluster Creation Modes Different Types of Tables in Databricks Iceberg Managed Tables in Databricks Magic Commands in Databricks Databricks Utilities Lakehouse Architecture Delta Lake in Depth Volumes Databricks Copy Into Autoloader Lakeflow Declarative Pipelines (Earlier called DLT) Implementing a Medallion Architecture Governance using Unity Catalog Lakeflow Connect Lakeflow Jobs Deployment – Databricks Asset Bundles 1 End to End Project The main attraction is not the curriculum, as it might look same everywhere. But the differentiating factor would be the depth of this content. I have already released 2 weeks, and my students are loving it. The ones who are doing it knows the depth in which we are learning it. If you learn Databricks in depth then you have a great advantage compared to others in this field. DM to know more details. There is a Launch offer going on as well! #databricks #bigdata #dataengineering

171

8 Comments

Powering the Future of Data Engineering! From scalable storage to real-time analytics, cloud platforms offer powerful data solutions. ✅ Storage & Management – Azure Data Lake, AWS S3, GCP Cloud Storage ✅ Processing & Pipelines – Azure Data Factory, AWS Glue, GCP Dataflow ✅ Big Data & Analytics – Azure Synapse, AWS Redshift, Google BigQuery ✅ Real-Time & Streaming – Azure Stream Analytics, AWS Kinesis, Pub/Sub ✅ Visualization & BI – Power BI, AWS QuickSight, Looker Below is a breakdown of key Data Engineering functions. Microsoft Azure Azure SQL Database + SELECT = Data Retrieval  Azure Data Factory + Pipelines = Data Movement  Azure Synapse Analytics + Distributed Queries = Big Data Processing  Azure Data Lake + Hierarchical Storage = Scalable Data Management  Azure Stream Analytics + Real-time Processing = Instant Data Insights  Azure Cosmos DB + NoSQL = Global-Scale Applications  Azure Databricks + ML Models = AI & Advanced Analytics  Azure Functions + Event-driven Processing = Server less Data Workflows  Azure Monitor + Log Analytics = System Performance Tracking  Azure Purview + Metadata Management = Data Governance & Lineage ️ Power BI + Dashboards = Data Visualization & Business Intelligence Amazon Web Services (AWS) AWS S3 + Data = Scalable Storage AWS Glue + ETL = Data Transformation AWS Athena + SQL = Serverless Querying AWS Redshift + Analytics = Data Warehousing AWS Kinesis + Streaming = Real-time Data Processing AWS Lambda + Data Pipelines = Serverless Orchestration AWS DynamoDB + NoSQL = Fast & Scalable Datastore AWS EMR + Spark = Big Data Processing AWS Step Functions + Workflow = Managed Data Orchestration AWS DMS + Migration = Seamless Database Replication AWS QuickSight + Visualization = Interactive Dashboards Google Cloud BigQuery + SELECT = Data Retrieval BigQuery + WHERE = Data Filtering  BigQuery + MATERIALIZED VIEWS = Precomputed Insights  Cloud Storage + GCS Buckets = Scalable Data Storage  Cloud Storage + Lifecycle Policies = Cost Optimization  Dataflow + Apache Beam = Scalable Data Processing  Dataflow + Streaming Pipelines = Real-Time Data Processing  Dataflow + Batch Pipelines = Scheduled Data Transformations  Pub/Sub + Topics & Subscriptions = Event-Driven Architecture  Pub/Sub + Message Retention = Reliable Messaging  Dataproc + Spark = Big Data Analytics  Dataproc + Hadoop = Distributed Data Processing  Cloud Composer + Apache Airflow = Orchestrating Data Pipelines  Data Fusion + GUI Pipelines = No-Code Data Integration  Looker + Dashboards = Data Visualization  Looker + Explorations = Self-Service BI 📙 Attaching a guide that compares Azure , AWS , and GCP Data Engineering cloud by aligning their core services with key engineering operations: 🚀 Join our 20K thriving Data Engineering Community ✨  https://lnkd.in/g-ZtB4Yf Follow Abhisek Sahu for more . Please Like, repost ✅, if you find them useful. #aws #azure #gcp #cloud #engineering #dataengineering #bigdata

453

37 Comments

Snowflake #Coco vs. Databricks Genie | Test 2 I Super simple data engineering task: "Create a scheduled task that runs every minute to insert 1,000 random rows from an existing table into a new table." It doesn't get easier than that. Here is how they handled it. Snowflake Cortex Code (#Coco) - Total Time: 3 minutes 14 seconds Coco breezed through the task with zero errors. It didn't just build the pipeline; it tested and verified it immediately. Why was it faster and more accurate? - Full Context: Deep, native access to the entire Snowflake platform. - Engineered Skills: It uses specialized Snowflake skills rather than relying on generic LLM knowledge. - SQL First Simplicity: Because Snowflake features are built with SQL access and RBAC (Role-Based Access Control) in mind, the agent can diagnose, grant permissions, and fix issues using simple SQL. It doesn't need to "hack" its way through Python scripts to make platform-specific API calls. Databricks Genie - Total Time: 9 minutes+ The experience was significantly more manual and prone to failure. - API Reliance: Genie struggled to modify job properties directly. It resorted to generating long Python applications to make API calls to Databricks. This felt no different than using a generic tool like Claude or Cursor. - Governance Gaps: The lack of fine-grained governance meant the agent had no way to switch roles, leading to an "all or nothing" security approach. - False Success : After five minutes, Genie claimed it was done but never actually tested the work. - Fix it loop - When I checked, the pipeline had errors. Asked the agent to fix it which it looped through many cycles of generic Python to access job logs to figure out the target table didn't even exist. - Token Waste: It went through multiple cycles of writing, failing, and rewriting generic Python code to diagnose issues that should have been simple SQL commands. This is what any other coding already does Cursor, Claude & etc. What makes Genie special? I don't know. Final Thoughts If Genie is the "new" GenAI coding agent, it isn't quite there yet. It feels like a minor upgrade rather than a brand-new feature. In its current state, you’d better hope your tokens are free because it wastes a lot of them trying to find the right path. Snowflake Cortex Code, on the other hand, is a massive leap forward. It acts as a true force multiplier, allowing the average user to build and maintain complex workflows & projects in a fraction of the time. Next up: What happens when the pipeline breaks and you need to know why?

632

14 Comments

30 most commonly asked Databricks interview questions (2025)— 1. What is Databricks, and how does it differ from traditional Spark clusters? 2. Explain the architecture of Databricks — how are clusters, jobs, and notebooks organized? 3. What are the different cluster types available in Databricks? 4. How does Databricks handle autoscaling and auto-termination? 5. What is the role of the Databricks Workspace? 6. What are Unity Catalog and its benefits in Databricks? 7. Explain the difference between Databricks SQL, Databricks Machine Learning, and Databricks Data Engineering. 8. What are DBFS and how is it different from cloud storage like S3 or ADLS? 9. What is the purpose of the Databricks Repos feature? 10. How can you integrate Databricks with Azure, AWS, or GCP? 11. What is Delta Lake, and why is it better than Parquet? 12. Explain ACID transactions in Delta Lake. 13. What is a Delta Log, and how does it maintain data consistency? 14. How do you perform Time Travel in Delta Lake? 15. What are MERGE INTO operations in Delta Lake? 16. How do you optimize Delta Lake performance (e.g., Z-Ordering, Data Skipping, Vacuum)? 17. Explain the difference between **bronze, silver, and gold** tables in a Databricks Lakehouse. 18. What are the **data retention** and **vacuuming policies** in Delta Lake? 19. How do you handle schema evolution and schema enforcement in Delta Lake? 20. Explain Change Data Capture (CDC) implementation in Databricks using Delta. 21. How do you tune Spark jobs in Databricks for performance optimization? 22. What are the key Spark configurations you usually tune in Databricks? 23. How do you use Adaptive Query Execution (AQE) in Databricks? 24. Explain how broadcast joins and skew joins are handled in Databricks. 25. How do you debug and monitor jobs in Databricks (e.g., Ganglia, Spark UI, Event Logs)? 26. What is Photon Engine, and how does it improve query performance? 27. Explain how Delta Live Tables (DLT) work in Databricks. 28. What are Databricks Jobs, and how do you schedule and monitor them? 29. What is **Databricks Autoloader, and how does it handle incremental data ingestion? 30. How do you implement end-to-end ETL pipelines in Databricks using Delta, Autoloader, and Workflows? : : : : Karthik K. Seekho Bigdata Institute 🔥 Let’s build your Data Engineering journey together! 📞 Ready to Upskill? Want to achieve the same results? Let’s get started: 📩 Call us directly at: 9989454737 📥 Join the info group here: https://lnkd.in/gkVevx5w 🌐 Visit: https://lnkd.in/gfDiU3sk

27

1 Comment

💡 Ever wondered why Spark is often called the “successor” to Hadoop MapReduce, but not to Hadoop itself? Here’s the catch: Hadoop is two things • HDFS → a distributed file system (think of it as a massive hard drive spread across many machines). • MapReduce → a processing engine that runs jobs on top of that storage. MapReduce was a breakthrough, but every step wrote intermediate results to disk. Imagine solving a math problem where after every single step, you had to write it down in a notebook, close it, and reopen it before the next step — accurate, but painfully slow. Spark changed that game. It introduced Resilient Distributed Datasets (RDDs) that live in memory (RAM) across the cluster. Now you could chain multiple operations without constantly hitting disk, making it 10–100x faster for iterative tasks like ML training or graph algorithms. So, while HDFS (Hadoop storage) is still widely used, MapReduce (Hadoop compute) is largely replaced by Spark. Together, they often coexist in modern big data pipelines. In my next post, I’ll break down Spark’s ecosystem, from SQL to MLlib and how it extends beyond just a faster MapReduce. #ApacheSpark #Hadoop #BigData #DataEngineering #Learning

39

Something I have observed while working in Databricks: Have you noticed how serverless clusters start up almost instantly, while job compute clusters take a while “configuring”? That’s because serverless keeps a pool of warm compute ready, while job clusters provision resources from scratch every time. Here’s what I think about it: Job Compute: ✅ Lower cost ✅ Supports PySpark ✅ Flexible configuration ⚠️ Slower startup time Serverless Warehouse: ⚡ Faster startup time ✅ Dedicated for SQL workloads ✅ Lower management overhead ⚠️ Doesn’t support PySpark ⚠️ More expensive (Photon acceleration) ⚠️ Less customization When to use: - Use Serverless for quick SQL analytics and low-maintenance needs. - Use Job Compute for PySpark-based, large, or configurable workloads. The right choice depends on what your pipeline values more. #databricks #serverless #pyspark #dataengineering #cloudcomputing

17

2 Comments

(Podcast)- Path to Data Engineering - Day 5 In this video, we explore essential components and challenges related to Data Lakes: Data and File Formats – Understanding common file types (CSV, Parquet, ORC) and their impact on performance Metadata Management – Why metadata is crucial for discoverability and query efficiency Partitioning – How partitioning improves query speed and scalability in large datasets Compaction – Techniques to optimize storage and reduce small file problems Limitations of Data Lakes: • The challenge of schema-on-read and inconsistent data structures • The lack of ACID transactions, and what that means for data reliability We wrap up with a practical conclusion, setting the stage for understanding how technologies like Delta Lake and Apache Iceberg emerged to address these gaps. For more content like this visit www.devtechie.com Full podcast series available here https://lnkd.in/gPjjnRzh https://lnkd.in/gSWE4s9R

1

AI is pushing teams toward a shallow definition of “work”. If an AI tool produces SQL that adds RLS policies to 8 tables, it is easy to conclude the work is “8 DDL changes.” I do not think that is a useful measure. Counting DDL changes is like counting lines of code. It measures activity rather than delivery or ownership. The real effort is usually before and after the DDL: - understanding the business requirement and the data flows behind it - choosing the approach and owning the trade-offs - validating what the AI produced for security and correctness - defining acceptance criteria so “done” is measurable (and survives production) AI compresses execution time. It does not compress decision-making time. It does not remove accountability. If you are building AI-first applications on PostgreSQL/Supabase, what do you use as your definition of “done”? Reply with your approach. I am collecting patterns that work.

11

💡 One of the most detailed and insightful trainings I’ve attended recently about pySpark – Datax Ultimate Bootcamp by Anurag Srivastava sir. The session provided deep clarity on Spark Executor Memory Management, including: 🔹 Executor Memory Management - breakdown of Reserved, User, Execution & Storage Memory. 🔹 Unified Memory Model - how Spark dynamically balances execution and storage requirements. 🔹 Off-Heap Memory - its role in reducing GC overhead and optimizing performance. - covered many more concepts. ✨ This training gave me a strong understanding of how Spark efficiently manages memory to handle large-scale data processing and why tuning these components is critical for performance. Grateful to Anurag Srivastava 🙏 for such a clear and practical walkthrough of Spark internals. #Dataengineer #ApacheSpark #DataEngineering #BigData #LearningJourney #Bootcamp #SparkInternals

45

1 Comment

Unity Catalog Update: What Changed & How It Impacts Us! Databricks just rolled out an important update related to governed tags in Unity Catalog. — What Changed : We can no longer drop a column that has governed tags assigned. The drop operation will fail until we remove those tags. This helps prevent accidental data leaks and keeps governance intact. — How to Fix It: Before dropping a column, remove the governed tags first: -- Step 1: Remove governed tag UNSET TAG ON COLUMN <catalog>.<schema>.<table>.<column> <tag_key>; -- Step 2: Drop the column ALTER TABLE <catalog>.<schema>.<table> DROP COLUMN <column>; -- Step 3: Vacuum to clear old data VACUUM <catalog>.<schema>.<table> RETAIN 0 HOURS; Real-Life Example: Let’s say we tagged a column email_id with a governed tag like PII=True. Now, if we try to drop this column directly, it’ll fail. We must first unset the tag, drop the column, and vacuum the table. Explanation This change enforces stricter governance on data with sensitive tags. It ensures we can’t remove tagged data without explicitly confirming it’s safe to do so… a small but crucial improvement for compliance and data protection. #Databricks #UnityCatalog #DataGovernance #DataEngineering

56

2 Comments

Big Data Series (Part 2/x): Deep Dive into HDFS In Part 1 of this series, we introduced Hadoop and its core components. This section provides a deeper look into HDFS (Hadoop Distributed File System), the storage layer responsible for reliable and scalable data persistence in Hadoop. HDFS is designed to store very large datasets across a cluster of machines. Instead of relying on a single disk or server, HDFS distributes data across multiple nodes, allowing the system to scale horizontally and tolerate failures. File Block Architecture - When a file is written to HDFS, it is automatically divided into fixed-size blocks. - Default block size: 64 MB (configurable) - Blocks are distributed across different DataNodes - Example: - Input file size: 200 MB - Block breakdown: - Block 1: 64 MB - Block 2: 64 MB - Block 3: 64 MB - Block 4: 8 MB This block-level distribution enables parallel data access and processing. - Role of the NameNode The NameNode is the central authority for HDFS metadata management. It maintains: - File-to-block mappings - Block-to-DataNode mappings - File permissions and configuration - Replication factor information The NameNode does not store actual data. Instead, it acts as a metadata service that allows clients to locate data efficiently. - DataNodes and Heartbeats DataNodes are responsible for storing actual data blocks. Their responsibilities include: - Reading and writing data blocks - Replicating blocks as instructed by the NameNode - Sending periodic heartbeat signals to indicate health and availability If a DataNode stops sending heartbeats, the NameNode marks it as unavailable and triggers re-replication. - Data Replication and Fault Tolerance To handle node failures, HDFS replicates each data block across multiple nodes. 1. Replication factor: Number of copies maintained per block Ensures data availability even if one or more DataNodes fail This replication mechanism is the foundation of Hadoop’s fault tolerance model. 2. Replica Placement Strategy (Rack Awareness) - HDFS uses a rack-aware placement strategy to optimize reliability and performance. The NameNode places replicas based on network topology 3. Typical default strategy: i) First replica placed on the local node ii) Second and third replicas placed on a different rack This approach minimizes network traffic while protecting against rack-level failures. iii) Clients always attempt to read data from the closest available replica. - HDFS Access Pattern Characteristics HDFS is optimized for specific workloads: Well-suited for: 1. Very large files (hundreds of MBs to petabytes) 2. Streaming and batch reads 3. Write-once, read-many (WORM) access patterns Not well-suited for: 1. Large numbers of small files 2. Low-latency random reads 3. Frequent writes or in-place file updates 4. Once a file is written to HDFS, it cannot be modified; it can only be appended or rewritten entirely. Data below shows Data Replication and working of HDFS architecture.

5

Handling Lakhs of Records with Spring Batch + S3 + MySQL In this video, I’ve demonstrated how to process large volumes of data using Spring Batch. ✅ Read data from multiple CSV files stored in an S3 bucket ✅ Process the data in chunks for optimized performance ✅ Store the processed records into a MySQL database This approach is ideal for batch-heavy applications where scalability and performance are key. 📹 Watch the full implementation here: https://lnkd.in/gwuFCdge #java #javascript #javafullstack #springboot #android #ios #flutter #reactnative #react #mumbai #delhi #pune #noida #hyderabad #chennai #kolkata #banglore #ai #artificialintelligence

8

I have written various blogs about the Lakehouse architecture with Apache Hudi, Apache Iceberg & Apache XTable (Incubating). Here are the 10 highly read blogs. ✅ Open Table Formats and the Open Data Lakehouse, In Perspective - https://lnkd.in/dhDC_hNP ✅ What is Clustering in an Open Data Lakehouse? https://lnkd.in/dvXsxNab ✅ How to Optimize Performance for Your Open Data Lakehouse - https://lnkd.in/dY9k9SUp ✅ Getting Started with Apache Flink SQL and Apache Iceberg - https://lnkd.in/dhpDT9DV ✅ What is Apache XTable - https://lnkd.in/dMBhfiVw ✅ Data Quality in Iceberg with Write-Audit-Publish (WAP) & Branching - https://lnkd.in/dk2EGuB8 ✅ Concurrency Control in Open Data Lakehouse - https://lnkd.in/dkbWyrFi ✅ What is a Data Lakehouse & How does it Work? https://lnkd.in/dS-Cixst ✅ Building Analytical Apps on the Lakehouse using Apache Hudi, Daft & Streamlit - https://lnkd.in/dbsRNunJ ✅ Run Apache XTable in AWS Lambda for background conversion of open table formats - https://lnkd.in/dSrgCCFt If you found them insightful, bookmark and share. Happy reading! #dataengineering #softwareengineering

70

1 Comment

Most of the snowflake courses are costly and pricing between as low as ₹ 3000 ($30+) to ₹ 30000 ($300+) at max. That is the entry barrier to learn the snowflake fundamentals for a beginner. I believe, basic learning should be free or affordable to everyone. To live up my own my belief - I have launched free fundamental course of #snowflake on #youtube. Why you should learn #snowflake ❓  • Snowflake is one of the fastest-growing enterprise data cloud companies.  • 11K+ enterprise customers globally.  • Strong penetration into Fortune/Forbes Global 2000 companies.  • AI adoption is accelerating rapidly via Cortex AI and partnerships with OpenAI & Anthropic.  • Revenue approaching $5B annually with ~30% YoY growth.  • Net revenue retention above 125% indicates extremely sticky enterprise adoption. 💰 Global Salary Range by Role-  • Snowflake AI Architect - $𝟭𝟲𝟬𝗞–$𝟯𝟬𝟬𝗞  • Snowflake Architect - $𝟭𝟰𝟬𝗞–$𝟮𝟱𝟬𝗞  • Snowflake Principal Engineer - $𝟭𝟴𝟬𝗞–$𝟯𝟱𝟬𝗞  • Snowflake Solution Engineer - $𝟭𝟱𝟬𝗞–$𝟮𝟴𝟬𝗞  • Snowflake Consultant Contractor  - $𝟴𝟬–$𝟭𝟴𝟬/𝗵𝗿 Course Link - https://lnkd.in/dvnTd5Ud Please share this with one who is preparing for the snowflake. #Snowflake #DataEngineering #CloudDataWarehouse #SQL #Snowpark #DataAnalytics #AI #MachineLearning #ETL #BigData #DataArchitect #SnowflakeTutorial #SnowflakeTraining #DataEngineer #CloudComputing #cortexai Snowflake | Practyc

19