Learn Data Engineering

🚨 Andreas Kretz and Altimate AI are going live tomorrow! Learn: ➡️ Why current AI agents fail in real data workflows ➡️ What context (schemas, lineage, dependencies) really means in practice and much more!

Clicking around in the cloud console is not building infrastructure. Become a skilled data engineer and learn how to do this properly. That means:   ✅ Writing Terraform ✅ Using Git to manage your infra ✅ Automating your deployments ✅ And versioning everything   That’s exactly what we’re doing in one of my Academy projects. 1️⃣ We define our Azure infrastructure in Terraform 2️⃣ CI/CD runs via Azure DevOps 3️⃣ One change to your code kicks off: → a new Docker image → a fresh Lambda → and the whole thing gets rolled out automatically   And honestly? It’s not that hard. If you know how to write SQL, you can learn Terraform, too. 💪   ⚠️ Dive right into this Azure project via the link in the comments!

What happens when you mix: 🐍 Python + 🐳 Docker +⚡ AWS Lambda + ⏰ Timeseries DB +🔌 API + 📈 Grafana ?   You get a fully deployable, scheduled ETL pipeline. One that grabs live weather data and pipes it into a time series database, then visualizes it automatically.   Here’s the flow: 1️⃣ You write a Python script to pull live data from a weather API 2️⃣ Package it into a Docker container 3️⃣ Push that container to AWS ECR 4️⃣ Deploy it on Lambda 5️⃣ Schedule it using EventBridge 6️⃣ Store the data in TDengine, a time series database 7️⃣ Visualize it all beautifully in Grafana 🥂 Done.   You’ve just built a real-time data system that updates itself. No cron jobs, no cloud console clicking.   This is one of those projects where people go: “Wait, you can actually do that with Lambda and a time series DB??”   Yes. You can. And you should.   Great for showing off real ETL skills, AWS experience, and your ability to build complete pipelines, not just scripts.   ⚠️ Dive right into it via the link in the comments!

In this video, I do a quick walkthrough of the new Amazon SageMaker provided by Amazon Web Services (AWS) and show how everything works in the new unified interface. 👉 https://lnkd.in/da-hsuKM We get into: ✅ how SageMaker brings data analytics, notebooks, AI, and ML together in one place ✅ accessing data from S3 and the AWS Glue Data Catalog ✅ working with notebooks using both Python and SQL ✅ using the built-in AI agent to generate code, and exploring data with charts You’ll also see visual ETL pipelines, query editing with Athena, model selection and deployment, inference endpoints, and how to work directly in JupyterLab or VS Code, all inside SageMaker. If you’re curious how Amazon SageMaker looks and feels today, this is a fast end-to-end overview. #dataengineering #aws #awspartner #sagemaker #amazonsagemaker

AI Agents are just REST API calls: Change my mind! Without APIs no AI Agents. Here are 5 things you actually need to master:   ✅ 𝗔𝗣𝗜 𝗙𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹𝘀: Understand what APIs are, how they work behind the scenes, and why REST is the go-to standard in most systems ✅ 𝗔𝗣𝗜 𝗗𝗲𝘀𝗶𝗴𝗻: Learn how to structure your endpoints, choose proper request/response formats, and write documentation others can actually use ✅ 𝗙𝗮𝘀𝘁𝗔𝗣𝗜 + 𝗣𝘆𝘁𝗵𝗼𝗻: Build high-performance APIs with a modern framework that’s simple but powerful ✅ 𝗔𝗣𝗜 𝗧𝗲𝘀𝘁𝗶𝗻𝗴: Use tools like Postman to make sure your APIs behave as expected before they hit production ✅ 𝗗𝗼𝗰𝗸𝗲𝗿 𝗗𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁: Containerize your API so it can run anywhere, reliably and at scale   If you’re working with data pipelines, backend services, or systems that need to integrate with others, API skills are a must have!   Well-designed APIs keep your systems flexible, scalable, and easier to maintain over time.   But many developers struggle with poor structure, lack of testing, or APIs that are hard to deploy and maintain.   Check out my Building APIs with FastAPI course: ⚠️ Find the full training link in the comments!   ⚠️ We also help with personal Coaching to learn Data Engineering. Link is in the comments as well 👍

Last year I was awarded a free one year subscription to Learn Data Engineering Data Academy and now I’m happy to share that I’ve obtained a new certification: Associate Data Engineer! Special thanks to Learn Data Engineering and Andreas Kretz for the opportunity to learn from the academy and step up my skills! It's a great place to learn with comprehensive content!

Check out our new FREE LAB 🤩 Topic: Orchestration!

Big news, everyone 😃