Pavan Kokkula

👩‍💻 Worried about data mismatches during Snowflake or Google BigQuery migration? 🚀 Meet Infometry Inc's INFOFISCUS CDWTA—your assurance engine for automated, no-code testing that guarantees trust, accuracy, and peace of mind. ➡️ Schema & row-level validation ➡️ Column-level data checks ➡️ On-premise deployment for full control ➡️ Built for enterprise-scale migrations Migrate smarter. Validate faster. Greater confidence. Learn more: https://lnkd.in/giuEit53 . . . #AgenticAI #NoCode #DataWarehouseTesting #ETLTesting #DataMigrationTesting #AutomatedTesting #SnowflakeMigration #GBQMigration #INFOFISCUS #Infometry

8

🚨 Spark Performance Tuning: The 10 Configs Every Data Engineer Should Know 🚀 💭 “Why is my Spark job so sloooow?” 👉 Sometimes, it’s not your data or code — it’s your configs. Here’s the cheat sheet every Data Engineer should keep handy (with a sprinkle of meme energy 💯): ⚡ Top 10 Spark Configs (that save you hours of debugging): 1️⃣ spark.sql.adaptive.enabled 💡 > Spark be like: “Don’t worry, I’ll optimize it for you… at runtime.” 2️⃣ spark.executor.memory 🧠 > Allocate too little → 💥 OutOfMemory. Allocate too much → 🚪 Goodbye cluster slots. 3️⃣ spark.sql.shuffle.partitions 🔀 > Default = 200. Real world = ❌. Tune it to your data size! 4️⃣ spark.executor.cores ⚡ > More cores = more parallelism. But balance it, or your executors will fight. 5️⃣ spark.sql.autoBroadcastJoinThreshold 📡 > Tiny table? Broadcast it. Big table? Don’t even try. 🚫 6️⃣ spark.num.executors 🔢 > The secret sauce for scaling. 7️⃣ spark.serializer 🏎️ > Java serializer = 🐢. Kryo serializer = 🚀. 8️⃣ spark.default.parallelism ⚖️ > RDD ops deserve fairness too. 9️⃣ spark.speculation 🏃 > Got slow stragglers? Spark says: “Let’s race them!” 🏁 🔟 spark.driver.memory 💻 > Because your driver also needs some love ❤️. 📌 Meme-worthy summary: “90% of Spark performance = knowing what knobs to twist.” 🔧 🔥 Now I’m curious → 👉 What’s the first Spark config YOU always tune before running a job? Drop your secret sauce in the comments 👇 #DataEngineering #ApacheSpark #BigData #PerformanceTuning #BigDatapedia #SparkTips

1

Optimizing the ETL Process in Snowflake: What Every Data Engineer Should Know Snowflake has redefined how modern data teams build scalable, high-performance ETL pipelines but many still under use its most powerful features. Here's how to leverage Snowflake to streamline your Extract, Transform, Load workflows and deliver faster, cleaner, and more reliable data. 1. Extraction Made Flexible : Snowflake integrates seamlessly with cloud storage (AWS S3, Azure Blob, GCP Storage) and ETL tools like Fivetran, Informatica, Matillion, and Airbyte. With Snowpipe, you can enable continuous ingestion to keep data fresh in real time. 2. Transform Using the Power of SQL: Instead of relying on external compute, Snowflake lets you transform data using its elastic warehouses. Features like Streams, Tasks, and Zero-Copy Cloning give you full agility to build automated transformation pipelines without operational overhead. 3. Load Data at Scale With Efficiency: Using COPY INTO, Snowflake handles large-scale batch loads with automatic tuning. Combined with micro-partitioning, the platform boosts query performance without any manual index management. 4. Governance & Observability Built In: Role-based access control, Time Travel, and robust lineage through partner tools ensure your ETL ecosystem stays compliant and transparent. 💡 Why It Matters: A well-architected Snowflake ETL process isn’t just about moving data it’s about accelerating analytics, enabling real-time insights, and improving decision-making across the organization. Want to know More: 👉 Click here: https://lnkd.in/d3tPFuqj #Snowflake #ETL #DataEngineering #CloudDataPlatform #DataAnalytics #DataPipeline #Snowpipe #SQL #ModernDataStack #DataTransformation #CloudComputing #Fivetran #Informatica #Matillion #Airbyte #BigData #DataWarehousing

1

Even the most controlled AP environments leave money on the table. Duplicate payments, missed credits, and pricing discrepancies add up across complex systems. AI is helping change that. In this blog, we break down how apexanalytix uses AI to: • Identify high-probability recovery opportunities • Improve supplier engagement with data-driven outreach • Automate statement reviews and follow-ups • Accelerate the entire audit process Read more: https://hubs.ly/Q04fq6P_0

17

Master the Hadoop Ecosystem — Fast! Want to break into Big Data ? This Hadoop Ecosystem Quick Handbook is your go-to resource! It condenses complex concepts into an easy-to-understand format — covering HDFS, MapReduce, YARN, Hive, Pig, HBase, Flume, Sqoop, and more. Whether you're a student, professional, or enthusiast, this gives you the clarity and confidence to navigate the Hadoop stack like a pro. Download, revise, and level up your skills — one tool at a time! #Hadoop #DataEngineering

7

At FabCon 2026, Azure Databricks is introducing new capabilities across Lakebase, Lakeflow, and Genie to make it easier to build data and AI applications on Azure. What’s new: • Azure Databricks Lakebase is now GA as the operational database for AI agents • Lakeflow Connect Free Tier allows you to ingest ~100 million records/workspace/day from many popular enterprise data sources • Databricks One Mobile enables access to AI/BI, dashboards, and Genie on-the-go • Genie and Genie Code expand AI-powered analytics and enable agentic workflows for data teams • Azure Databricks Excel add-in connects Excel directly to governed lakehouse data Together, these capabilities simplify how data is ingested, governed, and used to power analytics and AI on Azure. https://lnkd.in/gFp8pctk

1,427

23 Comments

𝐃𝐢𝐝 𝐘𝐨𝐮 𝐊𝐧𝐨𝐰? 🚀 Databricks is built on powerful open-source technologies like Apache Spark, Delta Lake, and MLflow — giving enterprises unmatched flexibility and scalability. At TGH Software Solutions Pvt. Ltd., we help organizations unlock the full potential of Databricks with open, future-proof architectures that avoid vendor lock-in, improve data agility, and reduce long-term operational costs. From architecture design to seamless integration and ongoing optimization, our Databricks experts ensure your data platform is ready for tomorrow’s innovation. 👉 Transform your data ecosystem with TGH’s Databricks specialists: https://lnkd.in/dgYVJNhn #Databricks #DeltaLake #ApacheSpark #MLflow #DataEngineering #TGH #IntegrationExperts #DigitalTransformation #Lakehouse #DataPlatform

13

SQL Server is built to handle thousands of concurrent requests, but that doesn’t mean every configuration setting should be adjusted. One setting that’s often misunderstood is Max Worker Threads. Changing it without understanding how SQL Server manages threads can create performance problems rather than fix them. In this blog, Srikanth Chivuklula, Senior Data Platform Engineer, explains: • What worker threads do in SQL Server • How the Max Worker Threads setting works • Why the default configuration is usually the right choice • When THREADPOOL waits indicate an issue • The rare scenarios where changing this setting may help Read the full blog here: https://lnkd.in/eUWBNsKU

14

💡 Snowflake CEO Sridhar Ramaswamy gave insights on the untapped potential of GPUs in data computation and why we are only in the "early innings" of how accelerated computing will transform the way we process enterprise data at scale. Watch the conversation from NVIDIA Live: Setting the Stage ➡️ https://nvda.ws/4qgBAkQ

45

4 Comments

🚀 New Blog - Synthetic Data Generation in Snowflake: A Breakthrough for Secure Data Sharing and Testing Snowflake now offers built-in synthetic data generation using the GENERATE_SYNTHETIC_DATA stored procedure — going beyond traditional masking to create statistically realistic datasets without exposing real data. In this blog, Senthil Kumar Thiyagarajan explores how this feature allows you to: ✅ Maintain schema and statistical distributions ✅ Preserve join relationships ✅ Share data safely across teams and environments ✅ Enable testing, validation, and collaboration All without compromising privacy or compliance. 🔗 Read the full blog: https://lnkd.in/g7KAMdQQ #Snowflake #Cittabase #SyntheticData #DataPrivacy #DataInnovation #SnowflakePartners

28

💡 From Data Pipelines to Data Products — A New Mindset for Architects At #DigitalArchitectConclave2025, Shankar Radhakrishnan from Tiger Analytics posed a powerful question: “Are you architecting for delivery or for consumption?” He emphasized that the future of data architecture lies in Data Products — treating data not as a project output, but as a product with its own lifecycle, customers, and measurable value. This shift calls for a new mindset: 🔹 From Builder → Product Manager: Start with the business decision before writing a single line of code. 🔹 From Silos → Federation: Design for interoperability and composability from day one. 🔹 From “Done” → “Evolving”: Build with observability, feedback loops, and continuous improvement. With AI as the key enabler, architects now have the tools to automate curation, enhance discoverability, and power intelligent data ecosystems. A compelling call for architects to step up as product leaders of the new data economy. 🎥 Watch the full session here: https://lnkd.in/gg_6aj6E NIIT Limited #DAC2025 #DigitalArchitectConclave #DataArchitecture #AI #Innovation #StackRoute #NIIT Pankaj J. | Vishnupriya Raghavan | Ramesh Padmanabhan | Anthony Gomes | Duraivelan D | Gaurav Baidyasen (GB) | Virendra Rao | Rajendra Joshi | Ruchi Goel | Amit Singh | Zeeshan S | Vishal Khandelwal | Surabhi Gopal | Wahhaj Ahmad | Anuska Rath | Gunjan B. | Varun Mahajan | Sajid Patel

10

Data teams are under pressure. • Snowflake costs are rising • ETL pipelines are fragile • Business users want real-time dashboards • Internal teams are overloaded If your organization relies on Snowflake and complex ETL workflows, stability and performance are no longer optional — they are mission-critical. At NTech1, we provide dedicated offshore ETL and Snowflake engineering support for companies that need scalable, production-grade data infrastructure. What we help with: ✔ Snowflake architecture & performance optimization ✔ ETL pipeline development (dbt, Airflow, Informatica, Talend, SSIS, Python) ✔ Data warehouse migration & modernization ✔ Cost optimization & query tuning ✔ Real-time & batch data integration ✔ Ongoing managed data platform support We operate as an extension of your internal data team — not just as a vendor. Why companies choose NTech1: • Senior engineers with strong architecture background • US time-zone alignment • Structured delivery & governance model • Enterprise-level experience without enterprise overhead If your Snowflake environment needs optimization, stabilization, or scale — let’s connect. 📩 Reach out directly at Sai@ntech1.com 🌐 www.NTech1.com #Snowflake #ETL #DataEngineering #OffshoreSupport #DataWarehouse #CloudData #Analytics #NTech1

1

Learn the crucial difference between platform and analytics engineers for a successful data platform. By Ashok Singamaneni and Gaurav Nanda (Databricks)

9