Prashant Kumar Pandey

PostgreSQL 18 is almost out. Two new game changing features: 🐘 Asynchronous I/O AIO 🐘 UUID7 AIO Introduction of a native asynchronous I/O (AIO) subsystem. This new architecture is designed to significantly increase I/O throughput and reduce latency by allowing the database to perform other work while waiting for data to be read from storage. The AIO feature allows PostgreSQL to issue read requests without blocking, which is especially beneficial for systems with slower, network-attached storage common in cloud environments. The initial implementation specifically targets read-heavy operations, including: ✔️ Sequential scans ✔️ Bitmap heap scans ✔️ Vacuums For databases running these types of workloads, this could translate into performance gains of 2-3 times. However, it's crucial to note that this first release focuses on reads, so write-intensive OLTP applications will not see a direct benefit just yet. UUID7 PostgreSQL 18 now includes native support for UUIDv7, a modern standard for generating globally unique and sortable identifiers. This is a game-changing feature for developers building distributed systems who need efficient, index-friendly primary keys. Traditional random UUIDs (like UUIDv4) are excellent for ensuring uniqueness but are terrible for database performance when used as primary keys. Because they are completely random, new entries are inserted at random locations within a B-tree index. This leads to poor data locality and frequent page splits, a costly operation that fragments the index and slows down writes and reads. UUIDv7 solves this problem by embedding a timestamp at the beginning of the identifier. This makes the UUIDs sequential and naturally ordered by creation time. When new UUIDv7 keys are inserted, they are always added to the end of the index, which is far more efficient. This approach dramatically improves caching and minimizes page splits, resulting in significantly better performance. #PostgreSQL #PostgreSQL18 #Database #DBA #DatabaseAdministrator #DevOps #SystemAdministrator #CloudComputing #DataEngineering #DatabaseManagement #ITInfrastructure #CloudDatabase #DatabasePerformance #postgres #aio #uuid7

362

2 Comments

Apache Parquet Vs New File Formats. There’s no denying the impact Apache Parquet has had on the analytics ecosystem. - Columnar layout - Strong compression & encoding - The de facto format in data lakes and warehouses for over a decade But there are also a lot of new things happening with our analytical workloads. ❌ Workloads are no longer just batch analytics - they involve AI pipelines, low-latency use cases, and hardware-accelerated compute ❌ Hardware is not only CPU but also GPU, from x86 to ARM/RISC-V, with wide SIMD instruction sets becoming the norm ❌ And performance bottlenecks are seen in decompression speed, memory pressure and lack of vectorized execution paths As a result, we’re seeing a wave of innovation around next-gen file formats. Efforts like: BtrBlocks – introduces cascaded compression with lightweight codecs Nimble (Meta) – optimized for fast scan & inference LanceDB – tailored for vector search and ML use cases And the past week I read about “Fastlanes” Here are some highlights from their VLDB paper ✅ Expression Encoding: Flexible chains of lightweight codecs (FFOR, DELTA, DICT, FSST) that outperform heavyweight compression like Zstd ✅ Multi-Column Compression: Exploits inter-column correlations (e.g. equality, one-to-one mappings) to go beyond traditional columnar compression ✅ Segmented Layout: Decompresses in small vectors (1024 values) rather than rowgroups, reducing memory pressure and improving cache efficiency ✅ Compressed Execution Support: Returns compressed vectors to engines like DuckDB/Velox for SIMD/GPU-friendly query execution Here are some numbers I picked up on Performance: - 800× faster random access than Parquet+Zstd - 41% better compression than Parquet+Snappy - 2% better compression than Parquet+Zstd (without using heavyweight codecs) - Decoding accelerates with AVX512 (up to 40% faster) I think we are going to see more of fit-for-purpose formats in the future as we tackle specific use cases around AI workloads and to target performance. Paper link in comments. #dataengineering #softwareengineering

377

13 Comments

Struggling to learn Cloud Data Engineering the right way? You’ve gone through countless tutorials. And now terms like: ⛔ETL vs ELT ⛔Lambda vs Kappa ⛔Lakehouse Architecture ⛔Medallion Layers & Microservices ⛔all start blending together. Here’s what I tell everyone Spend only 20–30% of your time consuming content. Spend 70–80% building real things. Why? Because creation brings clarity. Tutorials don’t stick until you apply what you learn. Start here: Build a simple cloud-native data pipeline. Use services like: ⛔Amazon S3 ⛔AWS Glue or Lambda ⛔Amazon Redshift ⛔Apache Airflow / MWAA To help you get started, I’ve shared an excellent guide by Deepa Vasanthkumar & Ahmad Mohiuddin (see attachment). It covers: ⛔ETL vs ELT ⛔Batch vs Streaming (Lambda/Kappa) ⛔Modern Lakehouse setups ⛔Medallion Architecture ⛔Microservices in the data stack Will it take time? Yes. Is it worth it? Absolutely. Once you’ve built something real, you’ll approach interviews with confidence and clarity. And if you found this helpful, give it a like or share it with someone who needs a nudge out of tutorial limbo. #DataEngineering #CloudComputing #AWS #ETL #CareerAdvice #LearnByBuilding #Lakehouse #DataCareers

151

5 Comments

etcd is a distributed key-value store used to coordinate state across nodes in a PostgreSQL cluster managed by Patroni. In a PostgreSQL setup aiming for 99.99% availability, etcd plays a crucial role. Here is why: 👉 Cluster State Source of Truth etcd holds the authoritative state of your PostgreSQL cluster. Patroni nodes read from and write to etcd to coordinate leader election, replica health, and failover decisions. Without it, there is no consensus—just confusion. 👉 Reliable Automatic Failover When the primary fails, Patroni does not guess. It asks etcd. If etcd confirms consensus, failover is triggered instantly and safely. No split-brain. No manual intervention. Just uptime. 👉 Enabler of 99.99% Availability You cannot hit four nines of uptime if failover depends on manual processes. etcd provides the consistency and speed needed for fully automated, fault-tolerant failover—keeping SLAs intact when it matters most. 👉 Lightweight and Secure etcd is designed for fast, distributed, and consistent reads/writes. It brings consensus-based decision making to PostgreSQL HA in a lightweight and secure package. 👉 Built for Scale As your application grows, etcd scales with you. It supports multi-node deployment and integrates tightly with cloud-native infrastructure. For technology leaders who prioritize keeping mission-critical systems online, avoiding costly downtime, and scaling smoothly, this is the architecture that delivers. If you are building for resilience with PostgreSQL, etcd is the cornerstone of availability. Read more here: https://lnkd.in/dwrdP3kn

85

2 Comments

Powering the Future of Data Engineering! From scalable storage to real-time analytics, cloud platforms offer powerful data solutions. ✅ Storage & Management – Azure Data Lake, AWS S3, GCP Cloud Storage ✅ Processing & Pipelines – Azure Data Factory, AWS Glue, GCP Dataflow ✅ Big Data & Analytics – Azure Synapse, AWS Redshift, Google BigQuery ✅ Real-Time & Streaming – Azure Stream Analytics, AWS Kinesis, Pub/Sub ✅ Visualization & BI – Power BI, AWS QuickSight, Looker Below is a breakdown of key Data Engineering functions. Microsoft Azure Azure SQL Database + SELECT = Data Retrieval Azure Data Factory + Pipelines = Data Movement Azure Synapse Analytics + Distributed Queries = Big Data Processing Azure Data Lake + Hierarchical Storage = Scalable Data Management Azure Stream Analytics + Real-time Processing = Instant Data Insights Azure Cosmos DB + NoSQL = Global-Scale Applications Azure Databricks + ML Models = AI & Advanced Analytics  Azure Functions + Event-driven Processing = Server less Data Workflows  Azure Monitor + Log Analytics = System Performance Tracking  Azure Purview + Metadata Management = Data Governance & Lineage ️ Power BI + Dashboards = Data Visualization & Business Intelligence Amazon Web Services (AWS) AWS S3 + Data = Scalable Storage AWS Glue + ETL = Data Transformation AWS Athena + SQL = Serverless Querying AWS Redshift + Analytics = Data Warehousing AWS Kinesis + Streaming = Real-time Data Processing AWS Lambda + Data Pipelines = Serverless Orchestration AWS DynamoDB + NoSQL = Fast & Scalable Datastore AWS Lake Formation + Governance = Secure Data Lakes AWS EMR + Spark = Big Data Processing AWS Step Functions + Workflow = Managed Data Orchestration AWS DMS + Migration = Seamless Database Replication AWS QuickSight + Visualization = Interactive Dashboards Google Cloud BigQuery + SELECT = Data Retrieval BigQuery + WHERE = Data Filtering  BigQuery + MATERIALIZED VIEWS = Precomputed Insights Cloud Storage + GCS Buckets = Scalable Data Storage Cloud Storage + Lifecycle Policies = Cost Optimization Dataflow + Apache Beam = Scalable Data Processing Dataflow + Streaming Pipelines = Real-Time Data Processing Dataflow + Batch Pipelines = Scheduled Data Transformations Pub/Sub + Topics & Subscriptions = Event-Driven Architecture Pub/Sub + Message Retention = Reliable Messaging Dataproc + Spark = Big Data Analytics Dataproc + Hadoop = Distributed Data Processing Cloud Composer + Apache Airflow = Orchestrating Data Pipelines Data Fusion + GUI Pipelines = No-Code Data Integration Looker + Dashboards = Data Visualization Looker + Explorations = Self-Service BI 📙 Attaching a guide that compares Azure , AWS , and GCP Data Engineering cloud by aligning their core services with key engineering operations: 🚀 Connect with 8K+ Data professionals for free data resources , Career, Regular Job Updates : https://lnkd.in/gy4R55Tj ♻️ Repost to help others grow #cloud #engineering #dataengineering #bigdata #hiring #career #jobs

989

43 Comments

**How to Track DDL Changes in Snowflake: New YouTube Video Alert** Snowflake provides a seamless way to capture DML (Data Manipulation Language) changes using STREAMS or CHANGES clause. But tracking DDL (Data Definition Language) changes such as schema modifications is not as straightforward. Unlike DML, Snowflake doesn’t offer a built-in function or feature explicitly designed to capture DDL changes like column additions, drops, or data type modifications. Recognizing this gap, I’ve created an in-depth YouTube video tutorial that walks you through a practical approach to monitoring and capturing these critical schema changes. The video covers: Methods to detect when new columns are added to a table or existing columns are dropped How to identify changes in data types for existing columns If you want to maintain a complete Audit of your schema changes then this video is must watch for you. Video link :: https://lnkd.in/ggCHqj_r Subscribe my channel for more Snowflake Real time scenarios. #snowflake #ddl #dataengineering #youtube

96

5 Comments

Data pipelines have evolved and you need to keep up! Here's what you need to know: 🔹 ETL (Extract, Transform, Load): Preprocesses data before loading to conserve storage and compute (common in the Hadoop era). 🔹 ELT (Extract, Load, Transform): Loads raw data directly into scalable cloud storage, transforming later for agility and flexibility. 🔹 Streaming: Processes data continuously, ideal for real-time applications like fraud detection or live dashboards. 🔹 Zero-ETL: Tightly integrates operational and analytical databases, eliminating explicit pipeline steps for faster, simpler data syncing. 🔹 Data Sharing: Grants direct query access without moving data, perfect for secure external collaboration or cross-departmental data sharing. Each has unique strengths and successful data teams often blend these patterns depending on their needs. SQLMesh is always the right answer 😉. Am I a corporate shill? Maybe. But I AM DEFINITELY an open source shill 👍 Ready to learn? Follow - Zach Wilson and his DataExpert.io Bootcamps Joe Reis and his Practical Data Discord Carly Taylor, MSc and her @carlytaylor Substack Madison Schott and her Learn Analytics Engineering Substack Tobias (Toby) Mao at Tobiko (creator of SQLMesh and SQLGlot) Lester Martin 🥑 at Starburst + all things Trino Software Foundation Who else? #dataengineering #sql #datatransformation

469

19 Comments

Your ML model isn’t failing - it’s your MLOps. Too many ML models die in Jupyter notebooks. Not because they’re bad but because ML ≠ production. MLOps is the missing link. Here’s the roadmap: → Programming & Engineering • Python scripting, APIs (FastAPI/Flask) • Git branching, pytest, CI/CD (GitHub Actions) • Docker for portability, load/A/B testing → Core ML & Deployment • Pipelines with PyTorch/sklearn • Package + serve via TorchServe / MLflow → Cloud & Infra • AWS SageMaker, Vertex AI, Azure ML • IaC with Terraform/AWS CDK • Serverless inference (Lambda) → Automation & Orchestration • Airflow, Kubeflow, Argo • Experiment tracking with MLflow/DVC → Monitoring & Experimentation • Feature stores (Feast), explainability (SHAP/LIME) • Privacy-first ML + edge deployment (TFLite) → Production & Scaling • Orchestrate with K8s/ECS • Scale inference with API gateways/service mesh • Auto-retraining pipelines → Collaboration & Mindset • Agile + DevOps practices • Clear docs + reproducible workflows MLOps isn’t “extra.”  It’s how ML survives, scales, and delivers real business impact. 👉DM me for career guidance/ Enterprise AI set up 👉Join the community to stay updated on new GenAI-AgenticAI advancements:- link in the comments section Follow Ujjyaini Mitra for more insights on Enterprise Gen AI

75

8 Comments

Claude, ChatGPT, Snowflake Intelligence, Cortex Analyst, Databricks Genie, Zenlytic, TextQL. Think about any AI agent you connect to your data. They all get questions from business users like: “What was the average number of daily active users in the last quarter?” These AI agent connects to Euno’s MCP server to retrieve metadata. That’s when Euno gives these agents the missing context they need to understand your data. ** Just a reminder that Euno is a metadata platform so we only access your metadata ** Euno runs a semantic search and looks for dashboards that use DAU (which is semantically equivalent to Daily Active Users). It filters for certified dashboards using the trust signals you’ve already set in Euno. In this example Euno finds two candidates: 1. DAU used in the User Engagement dashboards 2. DAU_NEW used in the Product Statistics dashboard Both dashboards are certified. But Euno also shows how often each has been used in the past 30 days, along with the owner or creator. With that information, the business user can confidently choose the right one and the AI agent can deliver the correct answer. The same applies to tools like Cursor. Let’s say you’re an engineer and ask: “Can you find me the Tableau dashboard for Customer Success?” Cursor will reach out to Euno’s AI assistant to get that BI context because Cursor is focused on dbt development and doesn’t have access to that layer. You can even go further and ask which dbt models power that dashboard and how those models are calculated. Euno feeds Cursor with that real-time context as you build. It’s not just for business users. It’s for developers too.

25

5 Comments

10 Resources, 10 Databricks concepts👇 1. Delta Lake Internals https://lnkd.in/gFzStZM6 2. Cluster Management & Autoscaling https://lnkd.in/g6sJvutG 3. Structured Streaming https://lnkd.in/gfVM-f2a 4. Performance Tuning & Caching https://lnkd.in/g-f2qN3U 5. Unity Catalog & Access Control https://lnkd.in/gwja8apn 6. Jobs API & Workflows https://lnkd.in/gF6Q67vM https://lnkd.in/gAunpBQG 7. Delta Live Tables (DLT) https://lnkd.in/gscqV3Qg 8. Photon & Runtime Optimizations https://lnkd.in/g6gQqitK 9. Delta Change Data Feed (CDF) in Streaming https://lnkd.in/giBZKUXh 10. Lakehouse ETL Design Patterns https://lnkd.in/gmyMpVpT What else you would add to this list ? 𝗖𝗼𝗻𝗻𝗲𝗰𝘁 𝟭:𝟭 𝗳𝗼𝗿 𝗰𝗮𝗿𝗲𝗲𝗿 𝗴𝘂𝗶𝗱𝗮𝗻𝗰𝗲 → https://lnkd.in/gH4DeYb4 𝗔𝗧𝗦 𝗢𝗽𝘁𝗶𝗺𝗶𝘀𝗲𝗱 𝗿𝗲𝘀𝘂𝗺𝗲 𝘁𝗲𝗺𝗽𝗹𝗮𝘁𝗲 → https://lnkd.in/g-iw7FaQ ♻️ Found this useful? Repost it!

206

4 Comments

One of the core problems Apache Iceberg set out to solve is - how do you query on TBs of data, fast. The entire structure of Iceberg metadata files and statistics aims to solve this exactly: - Reduce the number of calls to object storage - Scan (read) as little data as possible - the more filtering we can perform before we read the files, the better At the same time, Iceberg statistics can feel overwhelming at times, there are just too many options and settings to tune. I keep going back to this blog post about Apache Iceberg statistics when thinking about how it all comes into play.

117

8 Comments

⭕ 𝗘𝗡𝗗-𝗧𝗢-𝗘𝗡𝗗 𝗗𝗔𝗧𝗔 𝗘𝗡𝗚𝗜𝗡𝗘𝗘𝗥𝗜𝗡𝗚 𝗣𝗥𝗢𝗝𝗘𝗖𝗧 - 𝗣𝗬𝗦𝗣𝗔𝗥𝗞 𝘅 𝗗𝗕𝗧 ⭕ Want to learn how real-world data engineering projects are built? - I’ve created a 4+ hour long masterclass where we go from scratch to production-ready pipelines using PySpark & DBT. Here’s what you’ll master: ✅ Spark Streaming ✅ Modular Coding with Classes ✅ Incremental Load & Upserts ✅ Ephemeral Materialization ✅ DBT Models & Snapshots ✅ Star Schema Design 📍 Find the video link in the COMMENTS and start building today! ➡️ This project brings together the best practices in PySpark + DBT, preparing you for real-world projects as well as interviews. ♻️ HAPPY LEARNING ♻️ #Databricks #DatabricksMVP #dataengineering

284

10 Comments

🔹#Designing a Robust ETL Pipeline with Snowflake + dbt + Airflow #2025 ----------------------------------------------------------------------------------- . . . . . . As a Senior Data Engineer, one of my biggest learnings has been: 👉 ETL is not just about moving data — it’s about designing a scalable, reliable, and maintainable system. A recent project I worked on (Healthcare domain): Data Sources: EHR systems, IoT devices (patient monitoring), CSV logs in AWS S3. Ingestion: Used Snowpipe + COPY INTO for continuous ingestion into Snowflake. Transformation: Leveraged dbt for modular SQL models, version control, and documentation. Orchestration: Apache Airflow scheduled ingestion + dbt runs + QA checks. Visualization: Data served to Power BI for analytics dashboards. ✨ Key Best Practices I followed: Use staging layers (raw → clean → analytics) in Snowflake. Optimize performance with clustering & caching. Build idempotent pipelines (re-runs don’t duplicate data). Add data quality checks with dbt tests. Implement CI/CD with GitHub + dbt Cloud for versioning. 📌 Takeaway: A well-structured Snowflake ETL pipeline can handle billions of records efficiently, but success comes from combining the right tools (Snowflake + dbt + Airflow) with best practices. 💬 Question to You: ➡️ What’s your go-to stack for building modern ETL pipelines? #Snowflake #dbt #Airflow #ETL #DataEngineering #Cloud #SeniorDataEngineer

27

Building with RAG? (Reposting this blog again due to multiple requests) Here's the Complete Developer Stack to Know in 2025 RAG (Retrieval-Augmented Generation) is transforming how we build intelligent systems—but your architecture is only as strong as the tools you choose. This visual guide outlines the modern RAG stack across 8 key categories 👇 🔹 LLMs LLaMA 3.3, Gemini, Claude, and Phi-4 offer diverse strengths for different use cases. 🔹 Frameworks LangChain, LlamaIndex, Haystack, and others support modular orchestration and pipeline design. 🔹 Vector Databases Chroma, Qdrant, Weaviate, Milvus, Cassandra, Postgres, Opensearch, and more power fast, scalable retrieval. 🔹 Data Extraction Tools like Crawl4AI, MegaParser, FireCrawl, and Docling help parse and structure raw web and document data. 🔹 Open LLM Access Hugging Face, Ollama, Groq, Together AI make it easier to experiment, deploy, and scale. 🔹 Text Embeddings OpenAI, SBERT, and BAAI BGE remain essential choices for matching retrieval relevance with model performance. 🔹 Evaluation Giskard, Trulens, and Ragas offer tools to evaluate generation quality, trust, and reliability. 🧠 This stack is evolving rapidly. Choosing the right tools can be the difference between prototyping in a sandbox and deploying at scale. 💬 Which of these tools are you using or exploring in your RAG stack? Any favorites or new discoveries to share? #RAG #LLM #GenerativeAI #LangChain #VectorDatabases #AIStack #LLMDevelopment #OpenSourceAI #AIInfrastructure #MachineLearning #AI2025 #PromptEngineering #AIProductBuilders #DataEngineering

493

41 Comments