Vlad Gheorghe

🛠 From Solo Builder to Big Corp Data Engineer After a long break, I’m finally posting here again. Since March, I’ve been working as a Data Engineer at die Mobiliar. A big shift from the past two years, where I was self-employed, freelancing and building my own product, Athlog, a training collaboration app for coaches and athletes. The change from solo to a large organisation has been full of contrasts. A few things that stood out: 🔄 Speed vs. Stability Indie projects and small teams move fast, sometimes chaotic, but incredibly productive. You have an idea in the morning and a working prototype by the evening. In a big corporation like Mobiliar, the pace is noticeably slower. Not because the teams lack skill, but because of the many dependencies, processes, and approval layers that slow things down. Often, a team can’t solve problems independently because the solution sits in another department’s backlog. The upside? Systems are more stable, better documented, and designed to scale. The downside? It takes much more effort to maintain momentum and ship quickly. This will take me some time to get used too. 🧠 Wearing All the Hats vs. Focusing Deep Running your own projects means doing everything: code, UX, ops, support, and marketing. This was both a blessing and a curse: I learned a ton about a wide range of topics I constantly felt scattered, like I was scratching the surface of everything, but rarely going deep. Now, I get to focus on data engineering, and it feels good to dig in and specialise. 🤝 Autonomy vs. Alignment As a freelancer, I could make decisions instantly, but I also had to own all the risks. In a larger team, alignment takes more time, but it leads to more scalable, maintainable systems that others can trust and build on. 💬 Solo Thinking vs. Collective Knowledge One of the best parts of working in a bigger team: the shared knowledge and architectural discussions. I’m learning from colleagues every week, especially around designing robust data platforms. I’ll be sharing more here again, mostly around: Lessons and reflections from my journey as a Data Engineer System design and data architecture thinking Personal projects and indie experiments A bit of real life from time to time 😄 👉 If you're into data, building things, or making sense of messy systems, let’s connect. Curious: If you’ve made the jump from solo/small team to big company, what was the biggest shift for you?

50

I didn't master SQL. And I got to lead data teams of up to 50 people at places like Scout24 SE. We are obsessed with testing for technical tools in every data interview. But we forget the skills that actually drive change: → Translating data into a compelling business story. → Building a vision that inspires both engineers and executives. → Navigating politics to get a project funded. You can't SELECT * FROM business_impact; That query requires empathy, strategy, and communication. What's one non-technical skill that has been more valuable to your career than any coding language? 👇 #DataLeadership #CareerDevelopment #DataCareer #SoftSkills 🏃 Follow Dr. Markus for more unconventional takes on data leadership 💾 save this post for your next career conversation.

89

62 Kommentare

Even after 3+ years as a 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿, I’ve realized one hard truth: You don’t get to pick 𝗦𝗤𝗟 or 𝗣𝘆𝗦𝗽𝗮𝗿𝗸 → you need to master both. That’s exactly why this "SQL vs. PySpark Cheat Sheet" has been a game-changer for me. It’s not just a reference, it's a quick-access comparison guide built for working Data Engineers juggling big projects and tight timelines. Big shoutout to Bosscoder Academy for simplifying this 🙌 🔗 Explore their programs here: http://bit.ly/3UCsfXd Here’s what the doc covers: ✔️ Side-by-side breakdown of SQL and PySpark syntax ✔️ Real-world operations: joins, pivots, window functions, CTEs, DDL, and more ✔️ Quick refresher on string handling, date/time functions, and transformations ✔️ Supports faster prep for interviews and smoother context switching in daily work ✔️ Crafted for engineers working with big data in production If you're aiming for Data Engineering roles at product-based companies, or just want to sharpen your edge at work this cheat sheet keeps your logic crisp and context clear. And if you’re seriously aiming to level up your DE career while handling a full-time job, I’d highly recommend checking out Bosscoder’s structured programs. They’ve helped 2200+ professionals make the leap with support in: → Real-world data projects → System design & DE architecture → SQL + PySpark mastery → Resume help, mock interviews, and job switch support #dataengineering #pyspark #sql #careerswitch #bigdata #techgrowth #sponsoredpost

311

51 Kommentare

D̵a̵t̵a̵ 𝐏𝐥𝐚𝐭𝐟𝐨𝐫𝐦 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬! How the data hierarchy has changed... 5 years is enough time to really change or do a lot of things. 𝟐𝟎𝟐𝟎 𝐰𝐚𝐬 𝐭𝐡𝐞 𝐲𝐞𝐚𝐫 𝐨𝐟 𝐓𝐚𝐥𝐞𝐧𝐝/𝐇𝐚𝐝𝐨𝐨𝐩 Now it's Databricks + dbt + Cloud 𝐓𝐡𝐞 𝐢𝐏𝐡𝐨𝐧𝐞 𝟏𝟏 𝐰𝐚𝐬 𝐫𝐞𝐥𝐞𝐚𝐬𝐞𝐝 𝟓 𝐲𝐞𝐚𝐫𝐬 𝐚𝐠𝐨 The current hype is all about the iPhone 16 𝐁𝐚𝐜𝐤 𝐢𝐧 𝟐𝟎𝟐𝟎, 𝐢𝐭 𝐰𝐚𝐬 𝟑-𝐬𝐭𝐚𝐠𝐞 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰𝐬 Freelancers these days will simply say no, and pick another customer Clearly, a lot has changed. And where tech has significantly evolved, so has the set up of teams that adopt it. In a data team, you'd need an 𝐞𝐧𝐠𝐢𝐧𝐞𝐞𝐫 to build pipelines, an 𝐚𝐧𝐚𝐥𝐲𝐬𝐭 to build reports and run models, and an 𝐢𝐧𝐟𝐫𝐚 expert to keep the lights on. In steps the 𝑷𝒍𝒂𝒕𝒇𝒐𝒓𝒎 𝑬𝒏𝒈𝒊𝒏𝒆𝒆𝒓. These experts design, automate, and scale the entire data platform. 👉 They build self-service frameworks so analysts and scientists can move faster without waiting. 👉 They automate governance, quality, and cost control into the stack. 👉 They own the foundations of modern data products. 𝐖𝐡𝐲 𝐝𝐨𝐞𝐬 𝐭𝐡𝐢𝐬 𝐦𝐚𝐭𝐭𝐞𝐫 𝐢𝐧 𝟐𝟎𝟐𝟓? 💡 Companies chasing AI/ML at scale can’t succeed without stable, automated, and cost-efficient platforms. 💡 Talented data engineers are evolving into platform engineers because that’s where the impact (and career growth) is. 💡 Businesses that keep hiring the “old model” risk falling behind, stuck in firefighting mode rather than scaling. This results in needing to hire fewer pipeline coders and invest in Platform Engineers. Although, I don't think Cacheper has the right idea... #PlatformEngineers #Databricks #Talend #dbt

176

1 Kommentar

🚀🚀 I spent 10 hours learning Parquet. Here is the 10-minute summary: A few weeks ago, I had the chance to "refactor" my parquet article with the help of Kirill Bobrov, senior data engineer at Spotify. Here is the summary: ◉ Why do we need Parquet? ◉ How does it organize the data? ◉ Why does it implement the hybrid format? ◉ How do the Parquet's read/write processes look? ◉ How does it help with the OLAP workloads? ◉ Parquet Optimization & Best Practices 👉 You can check the full article here: https://lnkd.in/g-bj-tax ──────── My mom read my articles to support her son. Now, she can design a data architecture and write ETL scripts. Join 𝟭𝟯,𝟰𝟯𝟵 𝗿𝗲𝗮𝗱𝗲𝗿𝘀 to receive demystified data engineering knowledge weekly: https://lnkd.in/gDpy__uR #dataengineering #softwareengineering #parquet

680

17 Kommentare

I spent the last few weeks deep diving with PySpark,  and let me tell you—performance tuning is an art! Here are three things I actually implemented that turned sluggish jobs into fast pipelines:  1. Killed the Shuffling involved: Moved from wide to narrow transformations wherever possible.  Rewrote joins with broadcast joins for smaller tables—cut job time by 70%.  2. Partitioning Strategies: Repartitioned the data based on actual usage patterns, not guesswork.  Before: 1000+ partitions with tons of skew.  After: Balanced, purpose-driven partitioning = massive I/O reduction.  3. Cache Wisely when needed: Cached only what’s reused multiple times, and  released persistently stored data when done.  Result? Some of our longest-running Glue jobs (30+ mins) now finish in under 6 minutes.  Same data, same logic—just smarter execution. Why do this? Because everything in Cloud costs money! Glue is a very expensive AWS Service! The longer it runs, more it costs! So, tuning should be done! Period! Use this amazing doc by Shwetank Singh to learn Spark! If you've read so far, do LIKE the post 👍   𝐏.𝐒: Get a HIGH Paying Cloud Data Engineering Job: https://aws.sachin.cloud & Learn no-nonsense Cloud Data Engineering with 434+ others: 𝐖𝐡𝐚𝐭𝐬𝐚𝐩𝐩 group for notification : https://w.sachin.cloud (phone privacy ensured, you won't see others )

262

25 Kommentare

If you’re building agents, you’ve probably added “memory” already—RAG, long context, a vector DB, maybe a summary buffer—and still watched the agent drift, repeat mistakes, or forget the one constraint that mattered. I just published a short Substack article based on a recent arXiv survey: “Memory in the Age of AI Agents: A Survey — Forms, Functions and Dynamics” (arXiv:2512.13564). What I like about this paper is that it stops treating memory as a single feature and frames it as a system: Forms: how memory is represented (in-context, external stores, latent/internal, etc.) Functions: what it’s for (factual, experiential, working) Dynamics: how it evolves (formation → consolidation/forgetting → retrieval) In the Substack post I distilled the survey into a practical map you can use when designing long-horizon agents—what to store, how to keep it clean, and how not to pollute the context. 👉 If you want more builder-focused deep dives on agents and LLM engineering, subscribe to the newsletter. 🔗 Link to the article in the comments.

470

25 Kommentare

Why Data Warehouse Automation Still Matters in 2025 Even in the age of data lakes and lakehouses, data warehouse automation continues to prove its worth. My colleague and BARC analyst Florian Bigelmaier explains why this technology is still a game-changer for data teams. The three core benefits driving adoption are: 1. Efficiency gains: up to 80% time savings in modelling, testing and iterative processes in building, deploying and adapting data platforms >> Your data engineers can finally focus on the important topics instead of manual grunt work 2. Platform independence: metadata-driven approaches enable faster migration between different data warehouse platforms >> reducing vendor lock-in risks 3. Enhanced governance: comprehensive metadata management >> better auditability, data lineage tracking and version control These are intriguing market insights: - A $500 million tech market growing at 25% annually. - European vendors are playing an important role, with many coming from the Nordics, German speaking countries and the BeNeLux region. - The market is split between cloud-native newcomers and traditional on-premises or hybrid solutions. What does the future hold? >> AI integration is enhancing the capabilities of automation tools, making them even more powerful for modern data architectures. Here all his insights in this all-new Data Culture Podcast episode! Listen: https://hubs.li/Q03D5r3X0 Watch: https://hubs.li/Q03D5r4X0

25

We just released Dives — composable visualizations in MotherDuck, built by your favorite agents. Text-to-SQL works. The LLMs are really, really good. And now everyone has an army of agents in their terminal. Users aren't satisfied filing tickets for new reports — they want answers to O(n) questions, and they want them yesterday. Users can create Dives through the MotherDuck MCP Server using natural language. Dives are shareable, and exploration is instant thanks to MotherDuck's dual query execution. You can manage and update Dives with SQL, thanks to new SQL functions, and even check your Dives into a GitHub repo for version control and CI. We've already seen Tufte-style exec dashboards, Datadog dupes, an Excel emulator, and even Doom (of course). Jordan Tigani wrote about this moment in data and where we're headed. Read the blog: https://lnkd.in/gTtv56sQ Read the Dives docs: https://lnkd.in/g3ABSQty

169

10 Kommentare

Databricks published their approach to unified ML governance at the recent SIGMOD in Berlin providing an intuitive overview of how they approach this challenge: This is quite an interesting paper which shows how Unity Catalogue provides an open control plane that places tables, files, feature sets and MLflow-registered models in the same three-level namespace (catalog.schema.asset) while enforcing uniform row-, column- and path-level security via a one-asset-per-path rule and short-lived credential vending. This makes it possible for policies to hold whether Spark references the table name or another engine reads the raw object-store path. They also provide a glimpse in adoption and industry use, with roughly 100 M tables, 550 K volumes and 400 K models for 9 K customers, sustaining ~60 K metadata calls/s with latencies comparable to an on-cluster Hive metastore. Publication: https://lnkd.in/eu5F8nFu -- If you liked this article you can join 70,000+ practitioners for weekly tutorials, resources, OSS frameworks, and MLOps events across the machine learning ecosystem: https://lnkd.in/eRBQzVcA #ML #MachineLearning #ArtificialIntelligence #AI #MLOps #AIOps #DataOps #augmentedintelligence #deeplearning #privacy #kubernetes #datascience #python #bigdata

28

1 Kommentar

If a 200-year-old Swiss newsroom can modernize its entire data stack in a few months… 🇨🇭📰 What’s stopping federal agencies from doing the same? Newsroom moved to Snowflake’s AI Data Cloud — and everything changed: ⚡ 4x faster data refresh 💸 40% drop in processing costs 📈 13TB of unified data, refreshed hourly 🎯 Real-time insight into what readers care about most All running through one clean, scalable, AI-ready platform. Now imagine that same architecture powering your agency: 🏥 HHS spotting early health signals 🎖️ VA identifying at-risk veterans 🛡️ DOD sharing mission-critical insight across teams 📣 Agencies aligning messages to real-time citizen behavior Snowflake isn’t just a better data tool — it’s the connective tissue for smarter, faster, more responsive government. 📦 Replace legacy drag 🚀 Scale innovation 👥 Center the mission around people If NZZ can do it, so can we. #GovTech #DataCloud #Federal #Modernization #Snowflake #AI #DigitalGovernment #CX #Transformation #AIDataCloud #DataDrivenGovernment #SnowflakeGov #Media Read more about NZZ’s Snowflake journey here: https://lnkd.in/ehDurBTP

11

Kicking off a new series on PySpark ETL optimization!! Over the next few weeks, I’ll share practical tips for writing faster, more efficient ETL pipelines in PySpark, the kind of optimizations I use every day as a Data Engineer. Post #1 – Mastering persist() for Smarter Caching When you build multi-step transformations in Spark, it’s common to reuse the same intermediate DataFrame more than once. Without any caching, Spark will recompute that DataFrame every time it’s referenced, a big performance hit. persist() tells Spark to keep that DataFrame around so future actions don’t trigger the whole computation again. It’s like cache(), but more flexible, you can choose how and where the data is stored. Key storage levels you can pass: MEMORY_ONLY – Keep it in RAM (fastest, but may spill if memory is tight) MEMORY_AND_DISK – Store in memory, spill to disk if needed DISK_ONLY – Save directly to disk (useful for very large DataFrames) MEMORY_ONLY_SER – Same as MEMORY_ONLY but with serialized objects to save space MEMORY_AND_DISK_SER – Combination of the above It is important to say that calling persist() does nothing until you run an action to materialize the DataFrame. For example show(), count(), collect(), or a write operation will trigger the computation and actually populate the cache. In the next posts, I’ll cover other techniques like broadcast joins, partitioning strategies, and using checkpoints for complex DAGs!! Above a simple code example using persist.

258

31 Kommentare

Steps here👇the game has completely changed since 2022 - it’s not just leetcode anymore. Here’s what Viktoria Semaan shared that actually worked: ✅ Actually USE the platform (yes, she had to learn Databricks + Spark from scratch) ✅ Nail the coding assignment without ChatGPT crutches ✅ Come with CLEAR impact stories for leadership rounds ✅ Do your homework on every interviewer (LinkedIn stalking is real prep 😅) The technical bar is higher, but the human connection piece is everything. When you can tie your stories to their experience and show genuine interest in their work - that’s when you stand out. 2025 interviews hit different. Preparation is everything 💪 What’s the biggest interview challenge you’re facing right now? Drop it in the comments 👇 #techjobs #interviewtips

187

5 Kommentare