Building AWS Cloud Data Pipeline with AWS Lambda, Glue, and PostgreSQL

Most companies are quietly wasting thousands of dollars every year on their databases. 💸 During a recent technical practicum at Andrews University supervised by Roy Villafane, I compared AWS RDS (MySQL) vs AWS DynamoDB to find the best way to store 10 years of my university enrollment data. THE PROBLEM 🚗 Many organizations use instance-based databases (like RDS) to store rarely accessed historical data. It’s like renting a car by the day just to leave it in the garage. THE IDEA 🚕 What if we treated historical data like a taxi instead of a rental car? That’s essentially what a serverless approach with DynamoDB offers, you only pay when you actually use it. If nobody runs a report, the cost is $0.00. WHAT I BUILT * ETL with Python – Cleaned and merged 10 years of student data using pandas. * Dual-database pipeline – Automatically populated both RDS (MySQL) and DynamoDB in parallel. * Live benchmarking tool – Fired 1,000 requests at both systems to measure latency under load. THE RESULTS * RDS is strong for complex relationships and joins ✅ * DynamoDB’s key-value lookups were significantly faster for historical record retrieval, especially as request volume increased ⚡ REAL-WORLD CHALLENGES Along the way, I had to: * Configure SSL certificates (global-bundle.pem) for secure cloud connections 🔐 * Handle URL encoding for database credentials 😅 These “headaches” ended up being the best part, they felt like real production issues, not just academic exercises. This project confirmed how much I enjoy working with databases, data engineering, and cloud architecture, and how the right design choice can save a company both time and money. 👉 I’m curious: How is your team handling historical data today? Instance-based (RDS, EC2-hosted DBs)? Serverless / on-demand? Or maybe a data warehouse or lake? Let me know in the comments. I’d love to hear what’s working (or not) for you. #DataEngineering #AWS #Python #SQL #NoSQL #DynamoDB #CloudComputing #DatabasePerformance #DataAnalytics

This week, I had several DBA opportunities land in my inbox. Different stacks. Same expectation: Fix problems fast. Own the system. Don’t guess. A few stood out. - One was SQL Server-heavy, performance tuning, AOAG, replication, Azure. - Another leaned MySQL/Postgres, migrations, Linux, Python. Because in production, nobody cares what tool you use. They care how quickly you can diagnose why things are slowing down… and fix it before it escalates. And this is where the gap is becoming obvious. Some DBAs are still: Manually digging through logs Testing indexes blindly Reacting after performance drops Others have changed how they work. They’re using AI to: - Break down execution plans in seconds - Identify bottlenecks before users feel it - Simulate performance impact before deploying changes - Automate routine checks that used to take hours Not as a shortcut, as leverage. Because the real skill is no longer just writing queries. It’s understanding systems well enough to ask the right questions and using AI to get to answers faster. That’s what I’m seeing across roles now. SQL, MySQL, Postgres… those are expected. Cloud is expected. But speed of thinking? Speed of diagnosis? That’s becoming the real differentiator. The market isn’t announcing this shift loudly. But it’s already happening. And the people who adapt early won’t be competing for roles. They’ll be the ones teams reach out to first. So I’m curious: How are you currently using AI in your database workflow, if at all? #DatabaseAdministration #SQLServer #PostgreSQL #MySQL #AIinTech #DataEngineering #CloudComputing #TechCareers

PostgreSQL keeps evolving — and it's becoming increasingly relevant in AI-driven architectures. With the latest versions (including PostgreSQL 18) and the growing ecosystem around it, it's now much easier to build AI-powered features directly on top of a relational database. Extensions like pgvector are making it possible to: • store embeddings • run similarity searches • combine traditional SQL with semantic queries This is changing how we think about system design. Instead of introducing a separate vector database from day one, PostgreSQL can now handle a large part of the workload — especially in early or mid-scale systems. It’s not about replacing specialized tools, but about simplifying architecture and moving faster. I’ve been experimenting with this approach recently using Spring Boot + PostgreSQL, and it’s impressive how far you can go without adding new infrastructure. Curious to see how this space evolves. #PostgreSQL #AI #SoftwareArchitecture #Backend

𝑻𝒉𝒆 𝒎𝒐𝒎𝒆𝒏𝒕 𝑰 𝒓𝒆𝒂𝒍𝒊𝒛𝒆𝒅 𝒂 𝒅𝒂𝒕𝒂𝒃𝒂𝒔𝒆 𝒊𝒔 𝒇𝒂𝒓 𝒎𝒐𝒓𝒆 𝒕𝒉𝒂𝒏 𝒋𝒖𝒔𝒕 𝒕𝒂𝒃𝒍𝒆𝒔 𝒂𝒏𝒅 𝒔𝒕𝒐𝒓𝒂𝒈𝒆.... 🤔 💡 I used to think a database was just a place to store data. Learning MySQL, SQLite, or MongoDB felt good and enough. But when I started exploring PostgreSQL, my perspective changed completely. PostgreSQL is not just a database. It is a powerful data platform built with strong engineering principles and full ACID compliance. Features like: • ORDBMS architecture • MVCC for concurrency • Custom Data Types • JSON / JSONB support • pgvector for AI workloads • PgBouncer for connection pooling • WAL (Write Ahead Logging) • Rich Extensions ecosystem made me appreciate the depth and complexity of contemporary databases. This completely changed how I see databases. Now I am curious to explore how these systems work internally and keep feeding that curiosity. #PostgreSQL #DatabaseEngineering #Databases #BackendEngineering #SoftwareEngineering #DataEngineering #SystemDesign #DatabaseArchitecture #OpenSource #ACID #DistributedSystems #ScalableSystems #DevOps #CloudEngineering #AIInfrastructure #LearningJourney #TechCuriosity

Day 8/21: Moving to Persistent Data Storage Today's focus was addressing a critical limitation in my backend development: data persistence. Up until now, my application logic relied on server-side memory (RAM). This meant that any stored data was lost as soon as the server restarted or the process cycled. To build a reliable application, I have begun integrating a dedicated database layer to ensure data survives beyond the server's runtime. Technical Progress Data Lifecycle Analysis: Evaluated the differences between temporary variables and persistent storage, focusing on why a Database Management System (DBMS) is essential for production-ready applications. MongoDB Atlas Configuration: Provisioned a cloud-hosted NoSQL cluster. This moves the data layer from my local machine into a managed cloud environment, ensuring scalability and accessibility. Database Visualization: Utilized MongoDB Compass as a GUI to monitor and manage data collections, providing a clear view of how documents are structured within the database. Cloud Integration: Established a secure handshake between my local Express server and the Atlas cloud instance using connection strings and environment variables. Key Insight A backend architecture is only as robust as its data persistence. Shifting from local arrays to a cloud-hosted database like MongoDB is the first step in transforming a collection of scripts into a functional, data-driven application. Now that the infrastructure is connected, the next phase involves defining schemas to handle structured data entry. Sheryians Coding School Mentor: Ankur Prajapati #SheryiansCodingSchool #Cohort2 #21DaysChallenge #MongoDB #NodeJS #BackendDevelopment #CloudComputing #DatabaseArchitecture

🚨 Developers & Architects: If you still think SQL-first for every system, this MongoDB masterclass might change how you design scalable apps 👀 A FREE live online MongoDB Essentials program is open now on Hack2skill, and honestly this looks highly valuable for anyone building modern high-performance backend systems. We often talk about: → scalable microservices → flexible schema evolution → high-write workloads → product catalogs, events, analytics pipelines → document-heavy AI systems This is exactly where MongoDB’s document model and schema design patterns shine. 💡 What makes this worth attending? ✅ Transition from relational thinking → document-first architecture ✅ Learn embedding vs referencing for real-world modeling ✅ CRUD mastery with production-style querying ✅ Sorting, projection, filtering, logical operators ✅ Schema design patterns from MongoDB experts ✅ Official skill badge for LinkedIn ✅ Lucky rewards 🎁 (premium MongoDB cricket bats) The best part? The session is led by Mr. Andrew Morgan, a veteran voice in database architecture with 30+ years of experience. 🗓 Important Dates 📌 Registration deadline: 19 April 2026 📌 Live session: 22 April | 3 PM For developers working on: ✔️ FastAPI / Node.js backends ✔️ AI pipelines & vector workloads ✔️ event-driven microservices ✔️ high-scale product systems ✔️ real-time analytics …this is genuinely worth attending. I strongly feel understanding when to use relational vs document modeling is becoming a core engineering skill in 2026. If you're serious about backend architecture, don’t miss this. 🔗 Link: https://lnkd.in/dXvtiRHN #MongoDB #NoSQL #BackendEngineering #SystemDesign #DatabaseArchitecture #SoftwareEngineering #Developers #Microservices #FastAPI #AIEngineering #LinkedInLearning

Just picked up two new skills over last couple of days - Terraform & GitHub Actions - and used them to build an end-to-end data pipelines on AWS with CI/CD & Infrastructure as Code (IaC).  This project provisions a complete pipeline:   - Data pipeline : Raw --> Harmonized --> Curated data flow in S3 & RDS   - AWS Step Functions orchestrating Lambda + Glue ETL/Crawler jobs   - Glue Catalog + Athena for cataloging & querying   - All resources deployed with Terraform modules (networking, IAM, CloudWatch, S3, RDS, Glue, Step Functions).  Please take a look at my repo - https://lnkd.in/g4gsaPqQ What I found most exciting the ability to enable one-click deployment for both - code and infrastructure. I’m sharing this learning experience to gather feedback from the community on what could I do better. If you’ve worked on similar setups, I’d love to hear your thoughts on improvements - especially around enabling multi-environment support.  Happy Learning! #aws #terraform #github #githubactions #cicd #infrastructureasacode #iac #learningcontinues

I deleted a Docker container once and lost 2 days of database work. That was the day I learned about Volumes. 🐋 Most Docker beginners make this mistake — they run a database container, add data, restart or remove the container, and everything is gone. It's not a bug. Containers are designed to be ephemeral. But your data doesn't have to be. In my latest article, I cover everything you need to know about Docker Volumes: → Why containers lose data (and why that's by design) → What named volumes are and how to create them → The -v flag explained with real examples → Full PostgreSQL walkthrough — delete container, bring it back, data still there ✅ → Common mistakes beginners make with volumes → How volumes work in Docker Compose If you're running any database in Docker without a named volume right now — stop and read this first. 🔗 https://lnkd.in/eFZfXAam #Docker #DockerVolumes #DevOps #ContainerizedApps #CloudComputing #DockerCaptain #LearnDocker #DatabasePersistence #DockerTips #DockerForBeginners

Back to Basics: The Art of the SQL Join 📊 Even with 10+ years in the industry, I still find that a solid grasp of data relationships is the foundation of every great backend. I’ve been using this SQL Joins cheat sheet lately as a quick refresher while optimizing complex schemas in PostgreSQL and Azure SQL. In my recent work building patient data lakes and financial audit trails, choosing the right join—and knowing when to use a NULL check—can be the difference between a performant query and a system bottleneck. #JavaDevelopment #Python #PostgreSQL #SQL #DataModeling #BackendEngineer #Microservices #Azure #AWS #HealthTech #SaaS #TechCommunity #RemoteWork