Multitudes

Everyone’s talking about #AI-driven engineering productivity gains. But here’s the uncomfortable truth: most teams still struggle to measure them. I am excited to share my first LeadDev article where I break down why the hype isn’t matching reality! Based on brand new research from Multitudes and featuring unique insight from Lauren Peate and Sridhar Krishnan, VP of engineering, DevX and platforms, at LinkedIn! A few standout takeaways: 📏 Nearly all teams are adopting AI, yet most can’t clearly measure its impact despite growing pressure from leadership to prove results. 🎯 Engineering productivity is a moving target: traditional metrics like lines of code break down with AI. ✅ Measure broadly, not perfectly. Use simple, consistent metrics across usage, customer value, quality, and human impact. The real issue? Teams are optimizing for tasks (like code generation), while productivity is shaped by systems — workflows, collaboration, and decision-making. AI isn’t a magic multiplier. It exposes existing bottlenecks. If you’re serious about AI ROI, the question isn’t: 👉 “Are we using AI?” 👉 It’s: “Are we measuring what actually matters?” https://lnkd.in/eg7Z42qx

500+ engineers, 200+ leaders, and 15 months of data – all to get here: Our second AI impact whitepaper is now LIVE! 🎉 I don’t even want to tell you how many hours it takes a human to write a 30-page whitepaper. But not once was I tempted to have AI write this because – ironically? – using AI to write a research paper on AI’s impact feels especially wrong, if only to do justice to the 700+ people who shared their insights with us. I’ll have lots more to say about this paper later but for now, settle in with your favorite warm drink and get ready for a paper brimming with insights! To get you excited, our key takeaways were: 1️⃣ For more AI usage, build a learning organization. Learning organizations are ones that have systems that support continuous learning – something we especially need now given the pace of change of AI. 2️⃣ The desired AI outcomes are clear but measuring them is not. While organizations know what they want out of their AI tooling, many aren’t measuring outcomes – in part because it’s hard to know how to measure them. 3️⃣ Adapt processes, not just your tooling, to work with AI. Despite the importance of supporting people to get more from AI, the vast majority of the time, people focus on codebase changes rather than people or process ones. 4️⃣ Engineering roles are evolving. Engineers are expected to be more cross-functional, and with non-engineers building more software, there are additional demands on engineers to support these new builders. The most interesting part, of course, is how organizations are doing all the above, and so we’ve filled this paper with the rich examples that came up in our surveys and interviews. Also, a MASSIVE thanks to: Kelly Blincoe, Thomas Fritz, Brittany Johnson, MBA, Vivek Katial, and Nathen Harvey for the thoughtful feedback – on our draft survey and more recently on this whitepaper. You’ve made this research and whitepaper better, and any remaining issues are our own. And of course, HUGE thanks to Youxiang Lei for the analysis and writing, and to Laura Walker for the design wizardry. Whew, we did it!! Link to paper in comments – ungated, of course.

Since our AI impact measurement feature launched, we’ve been hard at work listening to user feedback and iterating. And now we have a TON of updates to share – a few highlights: 📊Intensity of Usage:  As we move from “are we using AI?” to “how much are we using AI?”, it becomes important to look at the depth of AI usage. That's what this feature does, considering cost, input tokens, and lines accepted. 💬Feedback insights: Human code reviews are the guardians of code health. To make sure you're getting enough high-quality human code reviews, you can see what types of feedback AI adopters are getting, and if the patterns of what they get feedback on have shifted. 📋AI survey With the pace of change in AI, it’s extra important to understand how our people are feeling. So that’s why we’re bringing in our very first survey to Multitudes. 🔭Open Telemetry: Most of you are using a lot of different AI tools. To make it faster to send us data from a wide range of sources, we now support Open Telemetry data, starting with Claude Code. 💲AI ROI calculator: Off the back of the launch of our free AI ROI calculator last week (link in comments), we now make it easy to calculate this from your Multitudes metrics. Tell us the methodology you’d like to use, and we’ll give you the metrics you need. But honestly, that’s just skimming the surface of all the features we’ve shipped recently – check out our docs or get in touch to see more! 

Talk long enough about AI tooling and someone will ask: Yeah, but what’s the ROI? I don’t talk about this often, but early in my career (between doing enough research to realize a PhD wasn’t for me and later discovering that startups are my happy place), I spent a few years doing strategy consulting. So if you want a financial model – perhaps, say, the ROI of AI tooling, then I’m your person. If you need the ROI of your AI tooling, then I have 2 resources for you: 1️⃣ *An ROI calculator for AI tooling* If you’re in a hurry, start here: Use our ROI calculator to estimate the financial impact of your AI tooling. I’ve given a few options for how you might calculate this, with the goal of allowing you to use metrics you already have to get the best possible estimate. 2️⃣ *A guide for measuring the ROI of AI tooling* To understand how the ROI calculation above works, read this. This guide combines my experience of using AI to build products with my previous stint doing financial models to share how I approach the ROI question. I also share key principles I’ve learned for building an ROI model well, based on common mistakes. In light of the massive Block layoffs last week, it is worth noting: Our model is based on the assumption that you’ll use the returns from AI tooling to deliver more value to your customers, not to cut staff. I’m interested in how we can use these tools to augment people, so that’s what I’ve built this model for. Links in comments. This is our first time doing a mini-calculator like this so I would love feedback – was this helpful? Should I do more data nerd + business mashups like this?

Tech leaders right now are caught in the middle, even more than usual: How can I set my developers up for success when there’s pressure coming down – around adoption or outcomes with AI – and also pressure coming up – from engineers who are concerned about risks to code quality, environmental issues, and what AI means for juniors roles? There’s no clear guidance from existing research yet either: 👟 Think AI will speed us up? The Microsoft Copilot paper says engineers will get 55.8% faster. 📉 Think AI isn’t what the hype says it is? There’s the METR study where engineers thought AI made them 20% faster but actually they were 19% slower. 👩 Worried about inequity and double standards in our AI rollouts? A Peking University study showed that women and “mature” engineers face a competence penalty for AI usage. And while there are existing surveys asking tech leaders about AI, they tend to focus on the more functional side – how much are you using it and what for, what productivity gain do you think you’re getting, and what’s the org’s investment in it. What existing research hasn’t covered as much is the human side of AI rollout – why leaders are rolling it out, the worries, the rollout challenges, and what it’s changed in how teams work together. Essentially, all the change management that is so hard but also so necessary for any big shift. So, drumroll please: Today we’re officially launching our AI impact survey for tech leaders. If you lead dev or dev-adjacent teams (eng, product, data, etc), then we want to hear from you! Survey link in the comments below 👇

📆 Next week – Tech Leader Chat, September 24/25 📆 You've heard us banging on about this before, but it's our next TLC! Presented by Aino Vonge Corry and our Lead Data Scientist Vivek Katial, we'll be talking about how to conduct human-centric, data-driven retrospectives. We know that retrospectives are important for engineering teams for a number of performance reasons. However, there is a gap in practice between the objective data and the subjective data used in retros to evaluate performance. There are also so many ways to conduct effective retros, so identifying what fits your team’s evaluative needs is a unique challenge for engineering leaders to consider. Vivek will talk through our research, and Aino (the author of Retrospectives Antipatterns) will share insights on: ⚖️ Common pitfalls that leaders encounter when running retrospectives 🚦 Why it’s important for teams to conduct data-driven retros 🤘 The impact of people problems on retros 🚂 How to maximize the benefits of retros for your team members Sign up on the event page in the comments below 👇Feel free to register even if you can't make it – we'll be recording the talk and will send you a copy afterwards!