ManojKumar Vadivel

A friend of mine interviewed at JPMorganChase for data engineer role (4 -5 )yrs exp3 below are combined few questions from both tech rounds. 1)How do you implement incremental data loads in ADF? 2)How do you handle failure and retries in ADF pipelines? 3)How do you parameterize pipelines in ADF? 4)How do you monitor and debug pipelines in ADF? 5)How do you orchestrate pipelines across multiple environments? 6)Have you used Azure Key Vault in ADF for securing credentials? 7)How do you optimize SQL queries for large datasets? 8)What is the difference between UNION and UNION ALL? 9)How do you handle NULL values in SQL joins and aggregations? 10)What is indexing, and how does it affect query performance? 11)How do you implement slowly changing dimensions (SCD) using SQL? 12How do you handle large data files in PySpark? 13)Explain the difference between repartition() and coalesce() in PySpark. 14)How do you perform window functions in PySpark? 15)What is lazy evaluation in Spark, and why is it important? 16)How do you optimize PySpark jobs for performance? 17)What is a broadcast join in PySpark, and when should it be used? 18)You have a daily batch pipeline that sometimes fails randomly. How would you debug and fix it? 19)How would you design a data pipeline that ingests files from ADLS Gen2 and loads them into a Delta Lake? 20)Describe how you would implement schema evolution in a PySpark-based pipeline. 21)How do you handle late-arriving data in your pipeline? 22) there was 2 codes that was on sql advance concepts like writing scd 2, a code on lead ,lag , row number use. #AzureDataEngineer #DataEngineerInterview#AzureDataFactory #PySpark#AzureDataEngineer#DataEngineerInterview#AzureDataFactory#PySpark #SQL#DataEngineering

247

10 Comments

Don’t get confused while learning PySpark. Understand these core concepts step by step to become truly proficient in PySpark. 𝗕𝗮𝘀𝗶𝗰𝘀 𝗼𝗳 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - PySpark Architecture - SparkContext and SparkSession - RDDs (Resilient Distributed Datasets) - DataFrames - Transformations and Actions - Lazy Evaluation 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗗𝗮𝘁𝗮𝗙𝗿𝗮𝗺𝗲𝘀: - Creating DataFrames - Reading Data from CSV, JSON, Parquet - DataFrame Operations - Filtering, Selecting, and Aggregating Data - Joins and Merging DataFrames - Working with Null Values 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗖𝗼𝗹𝘂𝗺𝗻 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Defining and Using UDFs (User Defined Functions) - Column Operations (Select, Rename, Drop) - Handling Complex Data Types (Array, Map) - Working with Dates and Timestamps 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗦𝗵𝘂𝗳𝗳𝗹𝗲 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻𝘀: - Understanding Partitions - Repartitioning and Coalescing - Managing Shuffle Operations - Optimizing Partition Sizes for Performance 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 𝗮𝗻𝗱 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗶𝗻𝗴 𝗗𝗮𝘁𝗮: - When to Cache or Persist - Memory vs Disk Caching - Checking Storage Levels 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 𝗪𝗶𝘁𝗵 𝗦𝗤𝗟: - Spark SQL Introduction - Creating Temp Views - Running SQL Queries - Optimizing SQL Queries with Catalyst Optimizer - Working with Hive Tables in PySpark 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗗𝗮𝘁𝗮 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Data Cleaning and Preparation - Handling Missing Values - Data Normalization and Transformation - Working with Categorical Data 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗧𝗼𝗽𝗶𝗰𝘀 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Broadcasting Variables - Accumulators - PySpark Window Functions - PySpark with Machine Learning (MLlib) - Working with Streaming Data (Spark Streaming) 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 𝗶𝗻 𝗣𝘆𝗦𝗽𝗮𝗿𝗸: - Understanding Job, Stage, and Task - Tungsten Execution Engine - Memory Management and Garbage Collection - Tuning Parallelism - Using Spark UI for Performance Monitoring Once you master these PySpark fundamentals, you won’t just write code — you’ll design efficient, scalable, and production-ready data pipelines that harness the full potential of Spark.

270

33 Comments

𝗛𝗮𝗱𝗼𝗼𝗽 𝗳𝗼𝗿 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 ⁣ If you're handling large-scale data or preparing for MAANG-level interviews → Hadoop still matters.⁣ Especially in legacy systems and enterprise-grade ETL.⁣ ⁣ 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗛𝗮𝗱𝗼𝗼𝗽? Hadoop is an open-source framework for distributed storage & parallel data processing.⁣ ⁣ 𝗪𝗵𝘆 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀 𝗨𝘀𝗲 𝗜𝘁: ⁣ → Stores terabytes to petabytes using HDFS → Processes data using MapReduce → Built for fault-tolerance with block replication → Integrates with tools like Hive, Pig, Spark, HBase → Supports enterprise-scale data pipelines⁣ 𝗕𝗮𝘀𝗶𝗰 𝗘𝘅𝗮𝗺𝗽𝗹𝗲: # Upload a CSV file to HDFS hdfs dfs -put data.csv /user/hadoop/data # Run a simple Hive query SELECT * FROM users WHERE age > 30; If you’re upskilling or switching to data engineering, I recommend Bosscoder Academy. Their hands-on guidance + mentorship makes it easier to get real-world ready. 👉 Check here: http://bit.ly/452taoH They’ve helped 2200+ professionals switch careers with a clear, practical curriculum, expert mentors, real-world projects, and full job support. #dataengineering #hadoop #workingprofessionals #data #sponsoredpost

171

21 Comments

𝗦𝘁𝗼𝗽 𝗪𝗿𝗶𝘁𝗶𝗻𝗴 𝗦𝗹𝗼𝘄 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗤𝘂𝗲𝗿𝗶𝗲𝘀 — 𝗛𝗲𝗿𝗲’𝘀 𝗛𝗼𝘄 𝘁𝗼 𝗦𝗽𝗲𝗲𝗱 𝗧𝗵𝗲𝗺 𝗨𝗽 🚀 Most Databricks users know how to 𝘄𝗿𝗶𝘁𝗲 a Spark job. But only a few know how to 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗲 it — and that’s where you stand out in interviews and on the job. From my experience, here are 𝗸𝗲𝘆 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗼𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝘁𝗲𝗰𝗵𝗻𝗶𝗾𝘂𝗲𝘀 you should master: 1️⃣ 𝗣𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝗶𝗻𝗴 & 𝗕𝘂𝗰𝗸𝗲𝘁𝗶𝗻𝗴 Efficient partitioning can drastically reduce shuffle, I/O, and query times. Bucketing helps with joins on large datasets — but only when applied smartly. 2️⃣ 𝗖𝗮𝗰𝗵𝗶𝗻𝗴 & 𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗲𝗻𝗰𝗲 Cache results when repeatedly accessed in the same job. But choose storage levels wisely — MEMORY\_ONLY vs MEMORY\_AND\_DISK can make or break performance. 3️⃣ 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲 𝗝𝗼𝗶𝗻𝘀 Use broadcast joins for small tables. Avoid shuffling massive datasets unnecessarily — and always inspect the physical plan. 4️⃣ 𝗗𝗲𝗹𝘁𝗮 𝗟𝗮𝗸𝗲 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀 Leverage 𝗭-𝗢𝗿𝗱𝗲𝗿𝗶𝗻𝗴 for better data skipping. Run `OPTIMIZE` and `VACUUM` to keep storage lean and queries fast. 5️⃣ 𝗔𝗱𝗮𝗽𝘁𝗶𝘃𝗲 𝗤𝘂𝗲𝗿𝘆 𝗘𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 (𝗔𝗤𝗘) Enable AQE to dynamically adjust join strategies and partition sizes at runtime. It’s one of the easiest wins for performance in Databricks SQL & Spark. 6️⃣ 𝗙𝗶𝗹𝗲 𝗦𝗶𝘇𝗲 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 Too many small files kill performance. Compact data into optimal file sizes (256–512 MB for most cases). 7️⃣ 𝗨𝘀𝗲 𝗘𝘅𝗽𝗹𝗮𝗶𝗻 𝗣𝗹𝗮𝗻𝘀 𝗥𝗲𝗹𝗶𝗴𝗶𝗼𝘂𝘀𝗹𝘆 Don’t guess where your query is slow — check the execution plan. Look for excessive shuffles, scans, and skew. 💡 𝗣𝗿𝗼 𝗧𝗶𝗽: Good engineers write working code. Great engineers write 𝗳𝗮𝘀𝘁, 𝗺𝗮𝗶𝗻𝘁𝗮𝗶𝗻𝗮𝗯𝗹𝗲, 𝗮𝗻𝗱 𝘀𝗰𝗮𝗹𝗮𝗯𝗹𝗲 code. If you’re preparing for a Data Engineering role or want to stand out at work — learn these techniques, practice them in projects, and measure performance before/after. 📌 Save this for your next Databricks project or interview. PDF Credit: Rahul Biswas 📌 𝗝𝗼𝗶𝗻 𝗺𝘆 𝗖𝗼𝗺𝗺𝘂𝗻𝗶𝘁𝘆: 🔗 𝗧𝗲𝗹𝗲𝗴𝗿𝗮𝗺: https://lnkd.in/gnA_WN76 🔗 𝗪𝗵𝗮𝘁𝘀𝗔𝗽𝗽: https://lnkd.in/gi2Mwzuq 🔗 𝗖𝗮𝗿𝗲𝗲𝗿 𝗚𝘂𝗶𝗱𝗮𝗻𝗰𝗲: https://lnkd.in/gFKyas-g Follow Aishwarya Pani for more 👋 #DataEngineering #Databricks #ApacheSpark #BigData #DeltaLake #Optimization #AzureDatabricks #InterviewPreparation

129

40 Comments

𝐑𝐞𝐚𝐥-𝐖𝐨𝐫𝐥𝐝 𝐐𝐮𝐞𝐬𝐭𝐢𝐨𝐧 ❓To process 10 GB of data using Spark. 👉 How many executors are required? 👉 How much memory should each executor have? 👉 How many cores are needed for maximum parallelism? ---------Step-by-Step Solution 𝟏. 𝐅𝐢𝐧𝐝 𝐧𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 𝐩𝐚𝐫𝐭𝐢𝐭𝐢𝐨𝐧𝐬 Default partition size = 128 MB Data size to process = 10 GB = 10,240 MB Number of partitions = 10,240 / 128 = 80 partitions 𝟐. 𝐅𝐢𝐧𝐝 𝐧𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 𝐜𝐨𝐫𝐞𝐬 Rule: 1 core = 1 partition parallelism Total cores required = 80 cores 𝟑. 𝐅𝐢𝐧𝐝 𝐧𝐮𝐦𝐛𝐞𝐫 𝐨𝐟 𝐞𝐱𝐞𝐜𝐮𝐭𝐨𝐫𝐬 Good practice: 5 cores per executor Number of executors = 80 / 5 = 16 executors 𝟒. 𝐅𝐢𝐧𝐝 𝐦𝐞𝐦𝐨𝐫𝐲 𝐩𝐞𝐫 𝐞𝐱𝐞𝐜𝐮𝐭𝐨𝐫 Formula: cores per executor × partition size = 5 × 128 MB = 640 MB (used just for execution) Recommended: multiply by 4 (to handle shuffle, overhead, caching, etc.) = 640 × 4 = ~2.6 GB per executor 𝐅𝐢𝐧𝐚𝐥 𝐀𝐧𝐬𝐰𝐞𝐫 Executors: 16, Cores per executor: 5, Total cores: 80, Memory per executor: ~2.6 GB This way, you can achieve maximum parallelism while avoiding out-of-memory issues. 💬 What’s your go-to strategy for tuning Spark jobs at scale? #BigData #Spark #DataEngineering #ApacheSpark #InterviewTips #Databricks #Azure #ETL #Pyspark #AWS #TechCareer #SparkTuning #CloudComputing ##LinkedInLearning

344

29 Comments

💥 𝗜𝗻𝗳𝗼𝘀𝘆𝘀 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 𝗬𝗼𝘂 𝗖𝗮𝗻’𝘁 𝗠𝗶𝘀𝘀 𝗶𝗻 𝟮𝟬𝟮𝟱 🚀 Preparing for Infosys interviews (13–17 LPA, 3+ YOE) Here are some of the real questions asked — the kind that truly test depth in SQL, ADF, Azure & Spark 👇 ✅ Job vs Interactive clusters in Databricks ✅ Metadata-driven pipelines in ADF (copy all tables) ✅ Data encryption in Azure SQL DB (TDE, Always Encrypted, TLS) ✅ SQL to find employees with no manager & 2nd highest salary ✅ PySpark deduplication (dropDuplicates, window functions) ✅ Incremental load in Databricks (watermarks, CDC, Delta MERGE) ✅ Role of Azure Key Vault in securing secrets ✅ Shuffling in Spark & its performance impact ✅ ADF triggers (schedule, tumbling, event-based) ✅ Best practices for scalable ADF pipelines 📌 Before your next interview, make sure you revise this quick guide — it will boost your confidence big time. Follow Kishore Mohammed for more interview preparation that actually works. #SQL #DataEngineer #Azure #ADF #Spark #InterviewPreparation

302

14 Comments

Hello Everyone, Scenario based interview questions for Azure Data Engineer role on delta tables part 2: Scenario 5: You have a streaming pipeline writing to a Delta table. The source starts sending a new column unexpectedly. What happens if the Delta table schema does not match? Answer: The write fails by default (Consistency). You must explicitly allow schema evolution using: .option("mergeSchema", "true") Or manually update the schema. Scenario 6: How would you audit what changed in a Delta table over the last 3 days? Answer: Use Time Travel with Delta Lake’s durability and versioning: spark.read.format("delta").option("timestampAsOf", "2025-07-23").load("/mnt/delta/sales") Scenario7: Your old pipeline writes data as raw Parquet. Sometimes, you find partial files or corrupted schema. How would Delta Lake and ACID transactions improve this? Answer: Delta Lake ensures Atomic, Consistent, and Durable writes. Partial files are never exposed, and schema is enforced. This avoids race conditions and schema drift common in file-based systems. Scenario8: You accidentally wrote incorrect data to your Delta table. How can you undo it? Answer: Use Time Travel and version restore: # Read previous version df = spark.read.format("delta").option("versionAsOf", 5).load("/mnt/delta/orders") Happy learning!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

101

4 Comments

📄 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗣𝗹𝗮𝘁𝗳𝗼𝗿𝗺 – 𝗗𝗲𝘁𝗮𝗶𝗹𝗲𝗱 𝗖𝗼𝗻𝗰𝗲𝗽𝘁𝘀 𝗜𝗻𝗱𝗲𝘅 I have compiled a structured reference covering the core components of the Databricks ecosystem, including:   • Core Platform Concepts   • Development & Collaboration   • Architecture & Storage (Lakehouse, Medallion, Delta)   • Tables & File Management   • Query Optimization & Streaming   • Security & Governance   • Real-Time Pipelines & Machine Learning   • Performance, Integrations & Marketplace This document is intended as a quick reference guide for professionals working with Databricks and the Lakehouse architecture. #Databricks #DataEngineering #DeltaLake #Lakehouse #BigData #MachineLearning

67

3 Comments

Hadoop vs Spark – Easy to Understand Hadoop (MapReduce + HDFS) → Works step by step, stores results on disk, good for very large data but slower. Spark (RDDs + Partitions) → Works in memory, processes much faster, supports both batch & real-time data. 👉 In short: Hadoop = reliable, but slower. Spark = modern, faster, real-time friendly. This image makes the difference clear in one glance #BigData #Hadoop #ApacheSpark #DataEngineering #DataProcessing #Learning

59

11 Comments

ON-DEMAND TOOLS | Data Engineering (High Demand in 2025) Apache Spark ⭐   Status: ON-DEMAND   Still the go-to engine for large-scale in-memory data processing and ETL workloads.  Snowflake ⭐⭐   Status: ON-DEMAND   Cloud-native data warehouse with massive enterprise adoption, often paired with dbt & Airflow.  Databricks ⭐⭐   Status: ON-DEMAND   Unified analytics & lakehouse platform built on Spark; dominates enterprise data stack discussions.  Apache Airflow ⭐⭐   Status: ON-DEMAND   Still the backbone for data orchestration; newer tools like Prefect & Dagster complement it.  dbt (Data Build Tool) ⭐⭐   Status: ON-DEMAND   Modern transformation layer; key skill for analytics engineers and SQL-based data modeling.  Apache Kafka ⭐⭐   Status: ON-DEMAND   Backbone for event streaming; used by most large-scale, real-time data systems globally.  BigQuery ⭐   Status: ON-DEMAND   Serverless, cost-efficient analytics engine widely adopted in modern cloud data architectures.  Apache Flink ⭐   Status: ON-DEMAND   Leading real-time stream processing framework; preferred over Storm for low-latency workloads.  Delta Lake / Apache Iceberg ⭐   Status: ON-DEMAND   Open table formats powering lakehouse architectures with ACID transactions and schema evolution.  --- ⚠️ OUTDATED OR DECLINING TOOLS Hadoop MapReduce ❌   Status: OUTDATED   Superseded by Spark and cloud-based engines; minimal new deployments.  Apache Hive ⚠️   Status: LEGACY   Still present in Hadoop clusters, but replaced by SQL-on-lake engines & modern warehouses.  Apache HBase ⚠️   Status: NICHE   Limited adoption beyond legacy Hadoop use cases.  Apache Storm ⚠️   Status: OUTDATED   Replaced by Flink & Spark Streaming for real-time workloads.  HDFS ⚠️   Status: DECLINING   Gradually replaced by cloud object storage (S3, GCS, ADLS).  --- 📊 KEY INSIGHTS (2025) 🔥 What’s Hot: - Cloud-native tools (Snowflake, Databricks, BigQuery)   - Modern orchestration (Airflow, Prefect)   - Stream processing (Kafka, Flink)   - Lakehouse tech (Delta Lake, Iceberg)   - Transformation tools (dbt) ⚠️ What’s Fading: - Traditional Hadoop stack   - MapReduce batch systems   - On-prem data clusters   - Hive & HBase for new projects  🎯 Career Tip: Focus on Spark, Airflow, dbt, Kafka, and Databricks/Snowflake — these dominate modern data engineering roles in 2025. Follow Gowtham SB #DataEngineering #BigData #DataPipeline #AI #Lakehouse #Spark #Airflow #Kafka #Snowflake #Databricks #dbt

288

10 Comments

Struggling with slow SQL queries? Here are 12 battle-tested tips to make your queries fly 🚀: 1️⃣ SELECT specific columns instead of SELECT * 2️⃣ Avoid using OR in JOIN conditions 3️⃣ Use CTEs instead of subqueries 4️⃣ Prefer Temp Tables over Table Variables 5️⃣ Create indexes on JOIN / filter columns 6️⃣ Use EXISTS instead of IN 7️⃣ Limit the use of DISTINCT 8️⃣ Consider Partitioning / Sharding for big data 9️⃣ Break down complex queries into smaller steps 🔟 Use UNION ALL instead of UNION 1️⃣1️⃣ Minimize functions in WHERE clause . 1️⃣2️⃣ Place the larger table on the LEFT side in joins . What else you will add in this list? Pro tip: Don’t forget to monitor query performance, analyze execution plans, and identify heavy nodes. Small rewrites often deliver massive gains. For Snowflake-specific optimization techniques, check out my blog: https://lnkd.in/gHMK7f2A #sql #optimizaton #tips

98

9 Comments