Built ConureDB, a distributed key-value database from scratch

99% of developers use PostgreSQL. But do they really get it? Sahn Lam made a post where he broke down the architecture and i really liked it, i wanted to add some valuable concepts i learned throughout using PostgresSQL on top of it. Knowing ACID and CAP theorem is table stakes. The engineers who truly stand out understand the architecture that makes Postgres so powerful. Here are the concepts that will level up your my database expertise: 1. It’s an Object-Relational Database. Forget just tables and rows. Postgres lets you define custom data types, inherit from tables, and store complex objects. It blurs the line between relational rigor and object-oriented flexibility. 2. TOAST: The Secret Sauce for Large Data. Ever stored a massive JSON blob? Postgres uses "The Oversized Attribute Storage Technique" (TOAST). When a row gets too big, it automatically compresses it (using its own PGLZ algorithm, similar to LZ77) or moves it to a separate "TOAST table." Your main table stays lean and mean. 3. MVCC: The Magic Behind Concurrency. How can readers never block writers and vice versa? Multi-Version Concurrency Control. Instead of overwriting data, Postgres creates new row versions for each transaction. Your query sees a consistent "snapshot" of the database from the moment it started. This is a game-changer for performance and isolation. 4. Write-Ahead Logging (WAL): The Guardian of Durability. No data left behind. Before any change hits the main data files, it's written to a sequential log (the WAL). This ensures durability after a crash and is the foundation for replication. Checkpoints then periodically flush data from memory to disk, making recovery fast and efficient. This isn't just academic knowledge. Understanding MVCC explains why you need VACUUM. Knowing TOAST helps you optimize storage. Grasping WAL is key to building resilient systems. This is the kind of deep, architectural understanding that separates good engineers from great ones. ----- Do you use Postgres or another Database Management? And why?

Mastering Database Reliability: How I Built a Fault-Tolerant Storage Layer in C++ When building backend systems, one lesson keeps resurfacing: your app is only as reliable as its data layer. Databases can fail:- due to network issues, downtime, or configuration errors , but your application shouldn’t crash because of it. I recently tackled this challenge by designing a fault-tolerant storage layer in C++, built around an interface I call IStorage. Why Data Reliability Matters In production systems, data loss or corruption is often more expensive than downtime. Whether you’re writing logs, authentication data, or configuration settings, you need mechanisms that gracefully handle database failures without blocking the entire application. Designing a Pluggable Architecture class IStorage : public IProductInventory, public ICategoryInventory { public: virtual ~IStorage() = default; }; Then I built multiple implementations, a database-backed one, and lightweight JSON and CSV versions. At runtime, the system automatically detects available storage options. If the database isn’t reachable, it falls back to file-based storage seamlessly. This design makes the application resilient and testable. I can run local tests on a JSON backend, then switch to PostgreSQL or MySQL in production, all without touching the business logic. Auto-Detection and Fallback Logic During initialization, the app tries to connect to the primary backend. If it fails, it logs the error and falls back automatically: inline std::shared_ptr<IStorage> createStorage() { auto db = std::make_shared<DatabaseStorage>( "dbname=bank user=postgres password=postgres host=localhost port=5432"); if (db->connect()) { db->initializeTablesFromFile("schema.sql"); return db; } else { return std::make_shared<FileStorage>(); } } The result? No user-facing errors, no data loss, and a much smoother recovery process. Lessons Learned 🚩 Abstraction is power. Interfaces make systems extensible and clean. 🚩 Design patterns matter. Applying Strategy and Factory patterns made switching backends effortless. 🚩 Reliability is a mindset. Building systems that expect failure is key to long-term stability. This project reminded me why I love backend engineering, blending architecture, resilience, and low-level performance. If you’ve worked on similar reliability patterns or fault-tolerant storage designs, I’d love to hear your approach. #CPlusPlus #SoftwareEngineering #BackendDevelopment #SystemDesign #Resilience #DatabaseEngineering

We just launched the Neon Data API, a PostgREST-compatible REST API for Postgres, reimplemented in Rust and embedded directly into the Neon platform: https://lnkd.in/gD-pth2Q PostgREST is one of the great projects of the Postgres ecosystem. It automatically turns your schema into a REST API with RLS-based access control. But PostgREST deployments usually run one process per project: this is great for individual apps, but at Neon’s scale (with many  thousands of active databases every day) this model gets compute-heavy and fragmented. So we built a new Data API that’s leaner, multi-tenant, and built to scale across tens of thousands of databases: Rust-based → lighter CPU/memory footprint Multi-tenant → one service safely serving many databases Part of the Neon platform → Every branch has its own API endpoint Compatible with scale-to-zero → DB compute idles when not in use, while RES is always available Flexible → works with Neon Auth or any JWKS provider The Neon Data API It’s still 100% PostgREST-compatible. Migration can be as simple as pointing your client to a new endpoint and updating auth. This is not only great for developers but also for agent platforms. With the Data API, you get a backend built on serverless Postgres as the foundation, complete with auth, branching, and scale-to-zero. This is just the start: more features to come 🔥

Agents don't behave like humans when they code. They fork prod, test 3 indexes in parallel, compare results, delete the forks, and report back - all in one conversation. That's not how humans use databases. It's not how we built databases either. So we spent the past year rethinking database infrastructure for this new reality. Today, we're launching Agentic Postgres - the first database designed from the ground up for agents. Three big technical problems we solved: 1. Making agents understand how to use Postgres well Our MCP server takes 10+ years of Postgres knowledge and bakes it into master prompts. Agents get high-level tools for schema design, query tuning, and migrations. Plus native semantic search over Postgres docs so they can find the right context while they're thinking. 2. Native retrieval without duct-taping a vector DB on the side Hybrid search built directly into Postgres. pgvectorscale for semantic search (higher throughput, better recall, lower latency than pgvector). pg_textsearch implements BM25 for ranked keyword search. In-memory for now, disk-based segments with compression coming soon. 3. Fast, zero-copy forks We built a copy-on-write block storage layer. Agents can spin up isolated environments in seconds - full production data copy, zero duplication. You only pay for blocks that change. Perfect for experiments, benchmarks, and parallel migrations. Under the hood: Fluid Storage. Disaggregated architecture with a distributed block store on local NVMe, storage proxy for copy-on-write volumes, and user-space block device driver. Looks like local disk to Postgres, scales like cloud. 100k+ IOPS per volume in testing. The interesting part isn't just the tech - it's what agents can do with it. Infrastructure that lets them work the way they actually work, not the way we wish they would. Try it yourself: curl -fsSL https://cli.tigerdata.com | sh tiger auth login tiger mcp install Free tier is live. All services run on Fluid Storage. Read the full technical details: https://lnkd.in/gU_swD4A

Tuesday Learning: Why Alex’s Go database strategy focuses on PostgreSQL optimisation over NoSQL complexity 📊 Working with Alexis Morin at Munimentum has taught me that most database “scaling” problems are actually database “optimisation” problems, especially for Go applications. What I’ve learned about Alex’s database approach: • PostgreSQL + Go’s pgx driver = exceptional performance without unnecessary NoSQL complexity. • Proper connection pooling can eliminate many scaling bottlenecks instantly. • Go’s type system helps prevent costly database errors at compile time. • Cost optimisation comes from efficiency, not expensive database technologies. The business translation: When Alex talks about “pgx driver optimisation and connection pooling,” I hear “handle more users with the same database infrastructure.” When he explains “vector database integration for AI workloads,” I see “reduce expensive OpenAI API costs through intelligent data caching and retrieval.” Real impact from Alex’s approach: His Go database optimisations typically achieve 3-5x query performance while reducing infrastructure costs by 40-60%. That’s sustainable scaling without database vendor lock-in. What I appreciate: Alex doesn't chase database trends. He optimizes proven technologies with Go’s strengths to deliver reliable, cost-effective performance. The best database strategy is often the boring one that actually works efficiently. Which database architecture decision has surprised you the most with its cost impact? #GoLang #DatabaseStrategy #PostgreSQL #TechnicalPartnerships #CostOptimisation

🗄️ DATABASE ARCHITECTURE FOR GO APPLICATIONS: Performance and Cost Optimisation Patterns Database design for Go applications requires strategic patterns that leverage Go’s strengths while minimizing infrastructure costs. Proper database architecture enables rapid scaling without proportional cost increases. Go-Optimised Database Patterns We start by ensuring best-in-class Go integration: • Integration Excellence: PostgreSQL with the pgx driver for superior performance and type safety. • Connection Control: Connection pooling optimization (pgxpool) to reduce database overhead and improve throughput. • Performance Levers: Prepared statements and query caching leveraging Go’s compile-time optimizations. • Reliable Evolution: Database migration strategies using golang-migrate for predictable schema evolution. Performance & Cost-Efficient Scaling Strategies Optimization is a strategic process, not a guessing game: • Query Mastery: SQL query tuning using EXPLAIN ANALYZE profiling and indexing based on actual Go application query patterns. • Scaling Smarter: Read replica implementation with Go’s context package for intelligent query routing. • Caching Power: Database query caching using Redis with Go’s excellent libraries (go-redis). • Resource Efficiency: Vertical scaling optimization before introducing complex horizontal overhead. Modern Data Architecture (AI/LLM Focused) The architecture must support future workloads: • AI Data: Vector database integration (e.g., pgvector with PostgreSQL) for AI applications. • Cost Management: Caching strategies for AI workloads to reduce expensive OpenAI API calls through data optimisation. • Data Modeling: Struct-based data modeling leveraging Go’s type system for compile-time safety. Operational Excellence Integrating development and operations for reliable database management: • Infrastructure as Code (IaC): Terraform for reliable database provisioning and configuration. • Full Visibility: Monitoring integration with Prometheus metrics for cost allocation and performance insights. Real-World Results: Go database optimisations typically achieve 3-5x query performance improvements while reducing database infrastructure costs by 40-60% through efficient resource utilization. Database architecture optimized for Go applications enables rapid scaling with predictable costs and exceptional performance characteristics. Which database performance challenges are limiting your Go application’s scalability and cost efficiency? #GoLang #DatabaseOptimisation #PostgreSQL #PerformanceOptimization #CostEfficiency

If you're struggling with high cloud bills, read this. 💸 We prove that strategic PostgreSQL optimisation with Go can deliver 40-60% cost reduction—without the complexity of expensive NoSQL migrations. Essential reading for every Go developer. See the full methodology from Alexis Morin and the team here! (Proud to be driving this strategy with Alex!) #DatabaseOptimisation #PostgreSQL #CostEfficiency #GoLang

Choosing a database shouldn’t feel like playing roulette — but it often does. Relational, document, graph, time-series… every database claims to be “the ultimate solution.” Until you start designing schemas and realize the trade-offs aren’t just technical — they’re architectural, financial, and sometimes emotional. I’ve been through that chaos more than once. We’d start a project thinking “let’s go NoSQL for flexibility” — then months later, we’re reinventing joins. Or go relational for consistency — then spend weeks scaling writes that NoSQL would’ve handled in minutes. Here’s what experience (and a few painful migrations) taught me: → Start with your data’s shape, not your favorite tech. If it’s relational, don’t force documents. → Model read/write patterns early. Your query frequency determines your pain points. → Plan for scaling and maintenance, not just speed. Ops overhead kills “fast” databases. → Test real-world workloads. Benchmarks mean nothing until you see how your data behaves. → Don’t fear hybrid architectures. Sometimes the best solution is combining two specialized tools. The right database isn’t the one trending on Hacker News — it’s the one that fits your problem and your team’s skill set. Databases define how your system thinks, grows, and fails. So choose like you’re picking a brain — not just storage. What’s the most painful database decision you’ve had to make (or undo)?

Building a Simple Data Platform that read Data from everywhere - Start with Docker Compose — one file, one command: docker-compose up - Three services: Polars (processing), DuckDB (query engine), Postgres (metadata tracking) - MinIO gives you local S3 — if blob writes fail here, they'll fail harder in production - Test your storage layer locally before AWS bills you for retry storms - Polars reads from anywhere — REST APIs, Kafka streams, S3 events, CSV files, Parquet - One library handles all formats — no separate tools for each source type - Don't separate batch and streaming infrastructure — use one read queue for both - Asyncio loop + Redis can handle "streaming" workloads without touching Kafka - APIs fail, rate limits trigger, schemas drift — build retry logic with exponential backoff - Dead letter queue catches everything else — review failures once daily, fix patterns, replay from blob - Store metadata in Postgres — source endpoint, last sync timestamp, schema version, row count - Every ingestion writes to MinIO first — raw data, immutable, timestamped - Polars processes 50GB+ CSVs without Spark — lazy evaluation loads what you need, when you need it - Transform once, write Parquet, query forever - DuckDB queries everything directly — join API responses with database dumps with stream data - No warehouse cluster, no spin-up time — just SQL that runs in milliseconds - Schema changes happen — Postgres tracks versions, Polars handles missing columns without breaking - Webhooks need fast responses — return HTTP 200 immediately, process async - Batch API requests — pull 1000 records per call instead of 1 - Dead letter queue stores failures — malformed JSON, missing fields, timeout errors - One docker-compose up runs: ingestion workers, blob storage, metadata tracking, query engine - Same setup works locally and in production — test the real architecture before deploy 📌 code https://lnkd.in/g77W3WFa here

When we say Lakehouse formats implements "ACID" properties - how do they implement? Transactions lie at the heart of reliable data systems. But the complexity of ensuring correctness across all possible scenarios (for example, failures, concurrency, or distributed operations) demands a robust framework. This is what "ACID" properties bring. A => Atomicity: A transaction is treated as a single unit. Either all operations succeed, or none do. C => Consistency: Transactions respect the rules and constraints of the system, preserving its integrity. I => Isolation: Transactions operate independently of one another, even if they are executed concurrently. D => Durability: Once committed, a transaction’s effects persist, even in the face of failures. While these properties have been enforced in database systems such as RDBMS (OLTP) and Data warehouses (OLAP), data lakes (such as object stores) have lacked such capabilities natively. Data lakes were inefficient in managing transactions, enforcing consistency, or reliably handling concurrent workloads. This often led to operational inefficiencies, corrupted data, and reprocessing overhead. The lack of ACID properties in a data lake has been a driving force for the emergence of the 'Lakehouse' architecture. Lakehouse architecture brings "storage engine" capabilities that provide database-like functionality atop distributed storage systems. These capabilities bring ACID properties into file systems such as Amazon S3 providing those reliability guarantees. All of these are made possible by open table formats such as Apache Iceberg, Apache Hudi & Delta Lake that form the backbone of a lakehouse architecture. Now, each format takes a slightly different approach to implementing these properties, reflecting their unique design choices and use cases. ✅ Iceberg uses the Catalog in the Metadata tree ✅ Hudi has the Timeline ✅ Delta uses transaction log But they have the same common goal - enforcing ACID! With my experience working with these 3 formats, I have been exploring how each implements ACID properties. In this detailed blog, I breakdown the internals and explain how they achieve ACID compliance. This should help data engineers/architects understand the nuances and serve as a guide for debugging issues & more! Link in comments. #dataengineering #softwareengineering