David Shayatovich

I sharded our database. Best decision ever. Until it wasn't. Last year, our write throughput was dying. Every user action—updates, state changes, activity logs—was slamming the same tables. Classic bottleneck. The solution felt obvious: shard by user_id. And honestly? It worked beautifully 💯 Write throughput went up 3x. Contention basically disappeared. Latency dropped hard. The team could breathe again. I thought we'd solved it. Then product evolved. Suddenly we needed users to collaborate. Share resources. Enforce global rules. Run workflows that touched multiple accounts at once. And that's when everything started breaking in weird ways. Simple queries now hit multiple shards. Transactions that used to be atomic became "eventually consistent" nightmares. I was writing compensating logic, retry handlers, background jobs to "fix up" state after the fact. That's when I realized , We didn't have a scale problem. We had Data Model problem. I'd sharded MutableSttate. Not events. I changed: - Instead of storing "current state" in shards, I made writes append-only. Every change became an event. Current state became a derived view—rebuilt from events. - This let me keep events in sharded storage (they have clear owners), run async pipelines for side-effects, and centralize only what needed to be global. The difference was night and day. Write spikes stopped hurting. Correctness got simpler. New features didn't need architecture rewrites. Sharding isn't a database trick. It's a domain decision. - If your boundaries are artificial (like "let's just split by user_id"), the system will expose that—eventually. - High write load ≠ shardable workload. - Fix the model first. Scale second. Yeah, event sourcing added complexity—schemas, replays, eventual consistency. But those problems were manageable. Cross-shard transactions? Not so much.

32

5 Comments

Things my clients never say: - "I really need a semantic layer" - "Which data warehouse tool has the most advanced features?" - "Our process is messed up because we don't have an orchestration tool" Things my clients do say: - "My reports don't line up. Where's a single place to look for answers?" - "I'm not sure where to invest my team's time and resources to improve this process" - "Which DMA will provide the most ROI for our marketing efforts?" As data people, we tend to care a lot about the tools and craft of data work. But our customers don't care about that. They're swarmed with problems that need reliable, useful data, and they often don't have the language to describe their challenges in "data language". Understanding the data question behind the business question is a superpower that allows data professionals to go from cost center to growth driver.

28

2 Comments

Remember Analytics Engineers is a 'role' invented by DBT, not a real one. Data Engineers, Data Analysts, Data Architect are truly real.

5

5 Comments

ETL isn’t dead. It just refuses to die, and that’s a good thing. The last few weeks I’ve been neck-deep in data science support work. I’m not a data scientist, but I’ve been helping another team build statistical models while leaning on ML.NET for scoring and grouping. Once you wrap your head around it, it’s surprisingly straightforward. The pilot tests we’ve been running have already given us some sharp insights. What caught me off guard was how much I’ve drifted back toward ETL processes instead of streaming. For years I was a die-hard “use events and streams when possible” evangelist. Kafka this, Kafka that, rewrite this service, consolidate these microservices, reduce complexity, the whole gospel. But the reality is, especially when you’re unfamiliar with the raw data, ETL is often the best way to validate, proof, and harden it before you even think about streaming or modeling. And let’s be clear: failed data equals failed output. I had my fair share of data quality issues last week, but cleaning and re-populating dimension and fact tables solved most of it faster than I expected, and it’s a tool I’ve seen far too many BI teams fail to embrace. So, mea culpa: ETL isn’t dying slowly. It’s here to stay for years. What frustrates me isn’t the technology, it’s the leadership vacuum. Too often the most junior person on the team has the curiosity and drive to actually make things work, while managers play “mystify the simple” and act like their jobs are unsolvable puzzles. For anyone newer to this space, the basics still matter: - Dimension tables are the context and descriptive attributes - Fact tables are the measurable events like transactions, counts, and scores - Stat tables are the derived metrics, aggregations, and model-ready features That foundation is what carries both statistical modeling and ML pipelines. Get it wrong and everything downstream is garbage. Get it right and even a simple model can shine. And hint: put them in their own schemas if you want to keep it all clean. ETL and stat tables aren’t sexy, but they actually get the job done, which is more than I can say for some of the folks I’ve seen put in charge of them. #DataScience #MachineLearning #ETL #DataEngineering #Analytics

8

What does it take to bring software engineering discipline to data transformations? Mike Huls shares a collection of powerful insights to help you escape messy SQL setups.

8

Love this post by Hussein Nasser. 💡 Using READPAST is such a clean way to handle worker queues without getting stuck in the trap of dead locks. ​It is worth mentioning that a lot of data engineers rely on NOLOCK or READ UNCOMMITTED when they see locked tables, and here it is important to mention the difference between these two approaches. ​READPAST vs. NOLOCK: - ​READPAST is the polite skip. If a row is locked, the query just jumps over it and keeps moving. You get only committed data, but you might miss a few rows that are currently being worked on. This is perfect to grab the next available job, which is the logic Hussein is talking about. - ​NOLOCK / READ UNCOMMITTED is the Wild West. It doesn't skip anything. It reads right through the locks, even if the data is mid-update. The problem with NOLOCK is that it’s fast, but it could cause dirty read. You may risk: - ​Ghost Data: Reading a row that someone else is inserting, but then they hit a rollback. You just processed data that technically never existed. - ​The mid-updated Row: Reading a record while it's being updated, so you get the new price but the old tax rate. - ​Double Counting: If a Page Split happens while you're scanning, you can actually count the same row twice or skip a whole chunk of data entirely. ​What do you think about these approaches, what is the best fit of each one? Data Engineers, let's discuss 🧠 ​#SQLServer #Database

5

Is this how pulling data from your Data Lake feels? Cribl Lake uses schema-on-need, so you can store telemetry in any format and shape it when you actually need it. No more guessing the perfect structure up front. You’re in control. Plus, the Lakehouse feature takes it further. Learn more: https://gag.gl/DhkuFW

1

When the conversation turns to Data Product delivery, most teams jump straight into Phase Zero: charters, ownership, RACI charts, etc. That's a mistake. Before any of that the enterprise must be prepared for the cultural shift. Let's call it Phase Negative One. Because Data Products aren't a project, they're a new standard operating procedure. This week's article, "Are You Even Ready to Be Thinking About Deploying Data Products?" explores what you need to do before beginning your Data Products initiative. https://lnkd.in/ejr3EfXz

2

Your dbt project just got smarter. Not a replacement. An extension. Here's what I mean: dbt is great at what it does. SQL transformations, testing, documentation. But it stops there. It doesn't know when your source data changes. It doesn't understand your full pipeline. It doesn't respond to incidents at 2 AM. What if your dbt models were part of something that did? That's what we've been building. Take your existing dbt project, bring it into Ascend, and suddenly: Your models run automatically when upstream data actually changes. Not on a schedule. Not after some orchestration layer polls and checks. Actually triggered by the data itself. Otto, our AI agent, understands your dbt models alongside everything else in your pipeline. It can debug issues, suggest optimizations, and generate new models that follow your existing patterns. When something breaks, you get intelligent triage that traces the problem back through your dbt models to the source. Not just "this model failed" but "this model failed because this column changed type in this upstream table." Smart Components extend beyond dbt's Simple and Incremental transforms. They track code fingerprints and data lineage to avoid unnecessary rebuilds. We've seen teams cut compute costs by 40-60% without changing their logic. The integration takes about 10 minutes. Point at your repo. Configure your data plane. Done. dbt isn't going anywhere. It's just getting more capable. Our very own Cody Peterson is holding an exciting hands-on lab this Wednesday -- come check it out @ https://lnkd.in/grPxMU-F #dbt #DataEngineering #AgenticDataEngineering

8

Training new data analysts: "Here's our data dictionary." *Hands over 53-page document that contradicts itself* "Here's how our systems work." *Shows diagram that looks like spaghetti having an existential crisis* "Here's our standard process." *Reveals 14 different ways to do the same thing* "Any questions?" The real question: How do we keep producing functioning analysts from this chaos? Welcome to the team, new person. May the odds be ever in your favor. #datahumor

102

13 Comments

Melissa Anino was stuck trying to get Viz in Tooltip to work. She posted a question on Next-Level Tableau and I created this short tutorial for her that I wanted to share with you. This is the type of support that the NLT Community gives each other. Most of the time I'm not the person answering the questions and this has helped foster the support I want people to feel they get when they join us.

50

5 Comments

Your data lives in S3 buckets you can’t navigate. Permissions buried in cryptic scripts. Every team has its own version of the truth. Governance? You’re running on hope. Then you hear about Databricks Unity Catalog promising order. But the chaos isn’t in the code—it’s in the culture. Here’s how we fix it. You start by unifying data into a single metastore. No more hunting for files. One source of truth. Next, you pick a Data Steward. The person who’ll keep this ship steady. Now, you build structure: catalogs, schemas, tables. Clean. Organized. Then the hard part: defining access. You map roles and groups. Give teams exactly what they need—and nothing more. Suddenly, your analysts stop arguing over data quality. Your security team stops sleeping at their desks. You connect every data source—internal, external. The catalog grows. You activate audit logs. Lineage appears. For the first time, you see exactly who touched what. But here’s the twist: the real resistance isn’t technical. It’s the data scientists who hate asking for permission. The engineers who called governance “bureaucracy.” You spend more time explaining 𝘸𝘩𝘺 than writing code. Still, you push forward. You train teams. You create processes that scale. And slowly, the chaos quiets. Now, when a compliance officer asks for an audit, you smile. When a new project launches, teams move faster—because they trust the data. Governance isn’t red tape. It’s the rails that let your data teams fly. If you’re drowning in data sprawl, start here. The hardest part isn’t the steps. It’s convincing your team this isn’t a cage—it’s a compass. Let’s build smarter data teams. One catalog at a time.

2

Jaya Gupta hits the nail on the head — and the devil really is in the details. Most of the agentic hype right now is superficial. The underlying data only tells you what happened — a binary, post-hoc read path. That’s not intelligence; it’s logging. Context is where the real value lives. It explains how and why decisions were made. If agents only see outcomes — the aftermath and effects — they’ll automate outputs, not judgment. That’s how you scale activity while missing the point entirely. You end up with workflows driven by low-calorie, shallow signals that look productive but erode decision quality over time. The unlock is being in the write path: capturing intent, trade-offs, constraints, and exceptions at the moment of decision. Without that, you’re just replaying logs with a nicer UI. This is exactly the gap that Mano Pillai and the team at HyperMindZ.ai are focused on — treating decision context as first-class infrastructure so agents learn from precedent, not just outcomes. Agents don’t need more data; they need governed context to reason, adapt, and earn trust. Would love to continue this conversation — feels like a great topic for a deeper article, white paper, or even a podcast discussion with folks who are building in this space.

2

3 Comments

New on the Basekick blog: What is a columnar database? A practitioner's guide to why columnar storage rewrote the analytics stack. covering history, mechanics, when to use one, when not to, and where the category is headed in 2026. Honest take: when columnar wins, when row stores still win, and what to ask any vendor before you commit your data to their format. Link in the comments.

6

1 Comment

I used to think the answer to data chaos was more dashboards. - more charts. - real-time updates. - cleaner visualizations. Turns out I was solving the wrong problem. Here’s what actually happens on most teams: → they open 3 reports to answer 1 question. → spend 20 minutes interpreting charts. → debate whether the numbers are right. The problem isn’t dashboards. It’s that business users are forced to interpret data at all. They should be able to ask: “What is our average sales cycle length?" And just get a clear answer. Not a 20-page report with a maze of filters in 6 different colours. AI will change this. Soon, everyone will make data-driven decisions, in seconds. But only if the data foundation is solid. Data science was the hottest job in 2020. In the next decade, it’ll be data engineering.

31

12 Comments

It's #TutorialThursday! In this video, Tableau/Alteryx Consultant Marc Tucker demonstrates how to use Dynamic Zone Visibility to build a 3-level map drill-down so users can seamlessly go from State to County to Zip Code. Check it out! 👉 https://ow.ly/NB6K50XtGIu Love learning new data skills? Subscribe to our YouTube channel for easy, helpful tutorials on Tableau, Alteryx, and more! #TutorialThursday #DynamicZoneVisibility #DataLearning

2

Most technical people do themselves a huge disservice. They focus only on writing the Python or SQL script… and forget to ask why. I’ve met IT professionals who’ve been at the same company for 10+ years, yet they still can’t explain how the company makes money. That’s not a technical gap. That’s a mindset gap. Don’t tell yourself “it’s not my job to think about money.” It is. We’re all in sales. The moment you start understanding the why behind the why, you stop putting yourself in a small box and start growing into the value creator your company actually needs.

47

10 Comments