Learn Data Engineering

Clicking around in the cloud console is not building infrastructure. Become a skilled data engineer and learn how to do this properly. That means:   ✅ Writing Terraform ✅ Using Git to manage your infra ✅ Automating your deployments ✅ And versioning everything   That’s exactly what we’re doing in one of my Academy projects. 1️⃣ We define our Azure infrastructure in Terraform 2️⃣ CI/CD runs via Azure DevOps 3️⃣ One change to your code kicks off: → a new Docker image → a fresh Lambda → and the whole thing gets rolled out automatically   And honestly? It’s not that hard. If you know how to write SQL, you can learn Terraform, too. 💪   ⚠️ Dive right into this Azure project via the link in the comments!

What happens when you mix: 🐍 Python + 🐳 Docker +⚡ AWS Lambda + ⏰ Timeseries DB +🔌 API + 📈 Grafana ?   You get a fully deployable, scheduled ETL pipeline. One that grabs live weather data and pipes it into a time series database, then visualizes it automatically.   Here’s the flow: 1️⃣ You write a Python script to pull live data from a weather API 2️⃣ Package it into a Docker container 3️⃣ Push that container to AWS ECR 4️⃣ Deploy it on Lambda 5️⃣ Schedule it using EventBridge 6️⃣ Store the data in TDengine, a time series database 7️⃣ Visualize it all beautifully in Grafana 🥂 Done.   You’ve just built a real-time data system that updates itself. No cron jobs, no cloud console clicking.   This is one of those projects where people go: “Wait, you can actually do that with Lambda and a time series DB??”   Yes. You can. And you should.   Great for showing off real ETL skills, AWS experience, and your ability to build complete pipelines, not just scripts.   ⚠️ Dive right into it via the link in the comments!

In this video, I do a quick walkthrough of the new Amazon SageMaker provided by Amazon Web Services (AWS) and show how everything works in the new unified interface. 👉 https://lnkd.in/da-hsuKM We get into: ✅ how SageMaker brings data analytics, notebooks, AI, and ML together in one place ✅ accessing data from S3 and the AWS Glue Data Catalog ✅ working with notebooks using both Python and SQL ✅ using the built-in AI agent to generate code, and exploring data with charts You’ll also see visual ETL pipelines, query editing with Athena, model selection and deployment, inference endpoints, and how to work directly in JupyterLab or VS Code, all inside SageMaker. If you’re curious how Amazon SageMaker looks and feels today, this is a fast end-to-end overview. #dataengineering #aws #awspartner #sagemaker #amazonsagemaker

AI Agents are just REST API calls: Change my mind! Without APIs no AI Agents. Here are 5 things you actually need to master:   ✅ 𝗔𝗣𝗜 𝗙𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹𝘀: Understand what APIs are, how they work behind the scenes, and why REST is the go-to standard in most systems ✅ 𝗔𝗣𝗜 𝗗𝗲𝘀𝗶𝗴𝗻: Learn how to structure your endpoints, choose proper request/response formats, and write documentation others can actually use ✅ 𝗙𝗮𝘀𝘁𝗔𝗣𝗜 + 𝗣𝘆𝘁𝗵𝗼𝗻: Build high-performance APIs with a modern framework that’s simple but powerful ✅ 𝗔𝗣𝗜 𝗧𝗲𝘀𝘁𝗶𝗻𝗴: Use tools like Postman to make sure your APIs behave as expected before they hit production ✅ 𝗗𝗼𝗰𝗸𝗲𝗿 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁: Containerize your API so it can run anywhere, reliably and at scale   If you’re working with data pipelines, backend services, or systems that need to integrate with others, API skills are a must have!   Well-designed APIs keep your systems flexible, scalable, and easier to maintain over time.   But many developers struggle with poor structure, lack of testing, or APIs that are hard to deploy and maintain.   Check out my Building APIs with FastAPI course: ⚠️ Find the full training link in the comments!   ⚠️ We also help with personal Coaching to learn Data Engineering. Link is in the comments as well 👍

Last year I was awarded a free one year subscription to Learn Data Engineering Data Academy and now I’m happy to share that I’ve obtained a new certification: Associate Data Engineer! Special thanks to Learn Data Engineering and Andreas Kretz for the opportunity to learn from the academy and step up my skills! It's a great place to learn with comprehensive content!

Check out our new FREE LAB 🤩 Topic: Orchestration!

Big news, everyone 😃

There’s a free webinar coming up on October 15: Best Practices for Writing ETL/ELT Pipelines hosted by Astronomer   Many data engineers still struggle with fragile DAGs, pipelines that break when data changes, or “hacks” to make things dynamic. If you’ve ever wanted your ETL/ELT pipelines to be more reliable, easier to maintain, and less painful to build then this session is for you.   🗓 October 15, 2025 | 11am ET / 4pm BST 🎙 Hosted by Tamara Fingerlin, Developer Advocate at Astronomer Save your spot here: https://lnkd.in/efqmXYDZ Top tip: register for it even if you can't join live. You'll get the recording afterwards 😉   You’ll learn: ✅ ETL vs. ELT and how to choose the right one for your use case ✅ Common Airflow DAG patterns that work well in production ✅ Key Airflow 3.0 features like dynamic task mapping, assets, and event-driven scheduling ✅ How to make DAGs dynamic in a scalable way (finally!)   I think this is going to be super helpful, especially for those of you working on complex pipelines or trying to scale your workflows across teams.   What's your biggest struggle when building ETL/ELT pipelines?