How Llms Impact Software Development

We know LLMs can substantially improve developer productivity. But the outcomes are not consistent. An extensive research review uncovers specific lessons on how best to use LLMs to amplify developer outcomes. 💡 Leverage LLMs for Improved Productivity. LLMs enable programmers to accomplish tasks faster, with studies reporting up to a 30% reduction in task completion times for routine coding activities. In one study, users completed 20% more tasks using LLM assistance compared to manual coding alone. However, these gains vary based on task complexity and user expertise; for complex tasks, time spent understanding LLM responses can offset productivity improvements. Tailored training can help users maximize these advantages. 🧠 Encourage Prompt Experimentation for Better Outputs. LLMs respond variably to phrasing and context, with studies showing that elaborated prompts led to 50% higher response accuracy compared to single-shot queries. For instance, users who refined prompts by breaking tasks into subtasks achieved superior outputs in 68% of cases. Organizations can build libraries of optimized prompts to standardize and enhance LLM usage across teams. 🔍 Balance LLM Use with Manual Effort. A hybrid approach—blending LLM responses with manual coding—was shown to improve solution quality in 75% of observed cases. For example, users often relied on LLMs to handle repetitive debugging tasks while manually reviewing complex algorithmic code. This strategy not only reduces cognitive load but also helps maintain the accuracy and reliability of final outputs. 📊 Tailor Metrics to Evaluate Human-AI Synergy. Metrics such as task completion rates, error counts, and code review times reveal the tangible impacts of LLMs. Studies found that LLM-assisted teams completed 25% more projects with 40% fewer errors compared to traditional methods. Pre- and post-test evaluations of users' learning showed a 30% improvement in conceptual understanding when LLMs were used effectively, highlighting the need for consistent performance benchmarking. 🚧 Mitigate Risks in LLM Use for Security. LLMs can inadvertently generate insecure code, with 20% of outputs in one study containing vulnerabilities like unchecked user inputs. However, when paired with automated code review tools, error rates dropped by 35%. To reduce risks, developers should combine LLMs with rigorous testing protocols and ensure their prompts explicitly address security considerations. 💡 Rethink Learning with LLMs. While LLMs improved learning outcomes in tasks requiring code comprehension by 32%, they sometimes hindered manual coding skill development, as seen in studies where post-LLM groups performed worse in syntax-based assessments. Educators can mitigate this by integrating LLMs into assignments that focus on problem-solving while requiring manual coding for foundational skills, ensuring balanced learning trajectories. Link to paper in comments.

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future ... The landscape of software engineering is rapidly evolving, driven by advancements in artificial intelligence. A recent research paper titled "From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges, and Future" sheds light on the transformative role of Large Language Models (LLMs) and their evolution into LLM-based agents. ✅ This paper is a must-read for anyone interested in the intersection of AI and software development.  👉 Key Insights from the Research  1. Understanding LLMs vs. LLM-based Agents The paper clarifies the distinction between LLMs and LLM-based agents: - LLMs are powerful tools for text generation and understanding, but they lack autonomy. - LLM-based agents integrate LLMs with external tools, enabling them to perform complex tasks autonomously, such as debugging and code refactoring.  2. Enhanced Capabilities in Software Engineering LLM-based agents demonstrate significant advantages: - They can autonomously debug, refactor, and generate tests, leading to increased efficiency and reduced human error. - These agents adapt to changing requirements, making them invaluable in dynamic software environments.  3. Key Areas of Application The research highlights several critical applications: - Requirement Engineering: Automating the capture and analysis of software requirements. - Code Generation: Streamlining the development process by generating code snippets from natural language descriptions. - Software Security: Enhancing security protocols and detecting vulnerabilities through proactive measures.  4. Challenges and Limitations Despite their promise, LLMs and LLM-based agents face challenges: - Context Length: Limited context can hinder their performance on extensive codebases. - Hallucinations: The risk of generating plausible but incorrect outputs necessitates human oversight to ensure accuracy.  5. Future Directions and Research Opportunities The paper discusses the need for: - Standardization and Benchmarking: Establishing unified standards to evaluate LLM-based agents effectively. - Exploration of AGI: Investigating the potential of LLM-based agents to approach Artificial General Intelligence, which could revolutionize software engineering practices.  👉 Conclusion This research paper provides a comprehensive overview of how LLMs and LLM-based agents are reshaping software engineering. By understanding their capabilities, applications, and limitations, we can better harness their potential to enhance productivity and innovation in the field. I encourage industry professionals to read this paper and engage in discussions about the future of AI in software engineering. What are your thoughts on the role of LLMs and LLM-based agents in our industry? Let's connect and explore this fascinating topic together.

Most companies are in between Software 1.0 and 2.0. Thanks to AI, Software 3.0 has arrived. (download 72 page slidedeck below) Andrej Karpathy’s recent talk at Y Combinator's AI Startup School introduces a concept that every tech executive should sit with: Software 3.0. Where Software 1.0 was about handcrafting logic, and Software 2.0 involved neural networks as black-box classifiers, Software 3.0 treats prompts as programs and LLMs as general-purpose computing substrates. This is the next substrate shift in software. The equivalent of mainframes → PCs → cloud → AI-native systems. First, let us review the Software 3.0 paradigm's 4 areas: 1. LLMs are the new operating systems, not just tools. They are: + Utilities (serving computation in the flow of work), + Fabs (mass-producing "digital artifacts" via generative interfaces), and OSes (abstracting complexity, orchestrating context, managing memory and interface). + The right way to view this is not "plug in an LLM." It is: what would a system look like if an LLM was your system's OS? 2. We’re entering the age of partial autonomy. Karpathy makes a compelling analogy to the Iron Man suit: + Augmentation: LLMs extend human capability (autocomplete, summarization, brainstorming). + Autonomy: LLMs act independently in constrained environments (agent loops, retrieval systems, workflow automation). + This leads to the concept of Autonomy Sliders — tuning systems from fully manual to semi-automated to agentic, depending on risk tolerance, verification requirements, and task criticality. 3. The Generator-Verifier loop is the new core of development. + Instead of "write → run → debug," think: Prompt → Generate → Verify → Refine + Shorter loops, faster iterations, and critical human-in-the-loop checkpoints. Reliability comes from verification, not perfection — a major shift for teams used to deterministic systems. 4. Architect for Agents, not just Users. + Your software doesn’t just serve end users anymore — it must now serve agents. These digital workers interact with your APIs, documentation, and UIs in fundamentally different ways. + Karpathy calls for a new class of developer experience: llms.txt instead of robots.txt + Agent-readable docs, schema-first interfaces, and fine-tuned orchestration layers. Some implications for AI implementations: A. Because of Software 3.0, enterprise architecture will evolve: traditional deterministic systems alongside generative, agentic infrastructure. B. AI Governance must span both. C. Investments in data pipelines, prompt systems, and verification workflows will be as important as microservices and DevOps were in the previous era. D. Your talent model must evolve: think AI Engineers not just Prompt Engineers, blending deep system knowledge with model-first programming. E. You’ll need a new playbook for build vs. integrate: when to wrap traditional software with LLMs vs. re-architect for Software 3.0 natively? What are your thoughts about Software 3.0?

LLMs are the new operating systems. And we’re all programming them in English. Software is undergoing a once-in-a-generation rewrite. Not just in what we build, but how. Andrej Karpathy’s recent talk at AI Startup School lays it out clearly: we’ve gone from → Software 1.0 (explicit logic) → Software 2.0 (neural nets with learned parameters) → to Software 3.0 (LLMs, programmable via English prompts). This isn’t just a clever metaphor. It’s a full-blown platform shift. “LLMs are utilities. LLMs are fabs. LLMs are operating systems.” And if that’s true, then today’s apps aren’t just software, they're the new UX layer for partial autonomy. Here’s what’s changing and what it means: 🔹 𝐏𝐫𝐨𝐦𝐩𝐭 = 𝐏𝐫𝐨𝐠𝐫𝐚𝐦 You don’t code anymore. You instruct. The syntax is natural language, the compiler is stochastic, and the runtime is probabilistic. Anyone who can think clearly can now build software. 🔹 𝐂𝐮𝐫𝐬𝐨𝐫 𝐚𝐧𝐝 𝐏𝐞𝐫𝐩𝐥𝐞𝐱𝐢𝐭𝐲 = 𝐄𝐚𝐫𝐥𝐲 𝐋𝐋𝐌-𝐧𝐚𝐭𝐢𝐯𝐞 𝐚𝐩𝐩𝐬 These apps don’t just call LLMs, they’re orchestrators. They manage context, layer GUIs for human verification, and offer autonomy sliders that let you decide how much control to cede. 🔹 𝐄𝐯𝐞𝐫𝐲 𝐚𝐩𝐩 𝐰𝐢𝐥𝐥 𝐡𝐚𝐯𝐞 𝐚𝐧 𝐚𝐮𝐭𝐨𝐧𝐨𝐦𝐲 𝐬𝐥𝐢𝐝𝐞𝐫 Like Iron Man suits, not Iron Man robots. We’re building augmentations, not agents. Yet. Keep the AI on a leash. Make the human-in-the-loop loop fast. 🔹 𝐖𝐞’𝐫𝐞 𝐛𝐚𝐜𝐤 𝐢𝐧 𝐭𝐡𝐞 1960𝐬 𝐨𝐟 𝐜𝐨𝐦𝐩𝐮𝐭𝐢𝐧𝐠 Time-sharing. Centralized compute. Batched queries. LLMs aren’t personal yet. We interact with them like dumb terminals plugged into a smart mainframe. That’ll change, but not tomorrow. 🔹 𝐃𝐨𝐜𝐬 𝐚𝐫𝐞 𝐟𝐨𝐫 𝐡𝐮𝐦𝐚𝐧𝐬. 𝐈𝐭’𝐬 𝐭𝐢𝐦𝐞 𝐭𝐨 𝐰𝐫𝐢𝐭𝐞 𝐟𝐨𝐫 𝐚𝐠𝐞𝐧𝐭𝐬. APIs were for programs. GUIs were for users. LLMs are a third interface type. We need llm.txt, Markdown-first docs, and agent-readable formats. Tools like DeepWiki and get.ingest are leading indicators. 🔹 𝐋𝐋𝐌𝐬 𝐡𝐚𝐯𝐞 𝐩𝐬𝐲𝐜𝐡𝐨𝐥𝐨𝐠𝐲 They're not machines. They simulate people. They’re savants with amnesia. Superhuman in some domains, clueless in others. We must learn to collaborate - without over-trusting. Why this matters for you: If you’re building software, stop thinking in code. Start thinking in agent affordances, prompt interfaces, and generation-verification loops. If you're an enterprise leader, don’t just “adopt AI.” Redesign your architecture to accommodate software that thinks, apps that adapt, and users that co-pilot. And if you’re in product, remember: partial autonomy will eat the GUI. The new UX isn't just visual. It's conversational, stochastic, and deeply probabilistic. “The future is less about programming computers, more about negotiating with them.” Build for people spirits. Design for GUIs and agents. And always, always audit the diff.

Is code still intellectual property in a world where AI can generate it on demand? The rise of LLMs is transforming the foundation of software development, challenging traditional notions of intellectual property. Historically, IP was straightforward to identify: it was the code itself. Companies invested months or years building proprietary algorithms and implementations that represented their crystallized knowledge of how to solve complex problems. Today, LLMs can generate sophisticated, production-quality code in minutes rather than months. While this code still has inherent value in its utility, its nature as intellectual property has fundamentally changed. When multiple developers can prompt an LLM to generate similar solutions, the code itself no longer represents unique intellectual capital. This capability not only democratizes software development but raises a profound question: If code can be generated so easily, where does the real value in software development now lie? The answer is that intellectual property in the LLM era isn't in the code itself, but in the empirical knowledge of what actually works in practice. While LLMs can generate theoretically perfect code, they can't capture the practical insights that only come from deploying solutions in the real world. Consider a complex integration between enterprise systems - an LLM might generate syntactically correct code, but only practical experience reveals which approaches handle edge cases, maintain performance at scale, and remain maintainable. The true IP becomes this accumulated knowledge of real-world implementation challenges. This shift fundamentally changes how we build software businesses. Success is no longer about accumulating proprietary code, but about systematizing practical knowledge of making LLM-generated solutions work reliably at scale. When an LLM can generate ten different valid approaches to a problem, the competitive advantage lies in knowing which approach will actually succeed in production, how to implement it robustly, and how to maintain it efficiently. This knowledge, unlike code, can't be easily replicated or generated. We're entering an era where the winners won't be those who write the best code—that's becoming commoditized. Instead, they'll be those who most effectively accumulate and apply practical knowledge about making LLM-generated solutions work in production. This represents a profound shift: while LLMs can generate multiple, seemingly valid solutions to any problem, their non-deterministic nature means each generation might produce different code. This uncertainty actually increases the value of implementation knowledge - success requires understanding not just one solution, but the patterns and principles that work reliably across many possible implementations. The true intellectual property is the battle-tested knowledge of how to navigate from AI-generated possibilities to production-ready solutions.

𝐂𝐭𝐫𝐥+𝐀𝐥𝐭+𝐃𝐞𝐥𝐞𝐭𝐞 𝐃𝐞𝐯 𝐂𝐚𝐫𝐞𝐞𝐫?⁣ LLMs Are Changing Software Engineering. ⁣ In recent months, I’ve been experimenting with LLMs across nearly every part of software engineering — prompting API scaffolds, debugging code, writing tests, generating infra files, even watching autonomous agents build full features end-to-end.⁣ The results are quite Impressive… and honestly, a little unsettling.⁣ ⁣ After 20+ years in this field, I’ve seen shifts — OOAD, mobile, cloud, Agile, DevOps, SaaS. Microservices...⁣ But this rise of LLMs and autonomous agents feels different.⁣ Not just faster — it’s fundamentally reshaping what it means to build.⁣ ⁣ I’m still figuring out what it means to be a developer in this new era — and how to guide others without pretending to have all the answers.⁣ We used to say software design is an art — a mix of intuition, structure, and elegance. But now AI generates not just code, but visual art, music, and prose. So maybe the art isn’t gone. It’s just changed. Maybe the real artistry now lies in how we prompt, guide, and critique the systems we build with... 🍺 Also on Constellar blog here: https://lnkd.in/gbv6YhWT #AI #LLMs #softwareengineering #developers #career #reflection

LLMs don’t change what software engineering is. They just make something obvious. The bottleneck was never writing code. Even before AI, typing was rarely the hard part. The real work has always been deciding what problem is worth solving, and what “good” actually looks like. It's turning ambiguity into decisions, making trade-offs under constraints and owning the outcome. LLMs are excellent at generating code and speeding up implementation. → But they don’t remove ambiguity. → They don’t decide what _not_ to build. → They don’t own design decisions, handle outages at 2AM, or take responsibility for long-term maintenance. That responsibility didn’t disappear. It stayed with the engineer. AI makes code cheaper. Judgement, clarity, and ownership just became more valuable. The bottleneck was never writing code. #softwareengineering #llm --- ➕ Follow Abdirahman Jama for software engineering tips

The biggest problem with LLM-based apps? It’s not the model. It’s not the framework. It’s not even the prompt. It’s everything around them. We’re not just calling an API. We’re integrating a stochastic system into a world that expects deterministic behavior. 𝗟𝗟𝗠𝘀 𝗮𝗿𝗲 𝗻𝗼𝘁 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰 𝗰𝗼𝗺𝗽𝗼𝗻𝗲𝗻𝘁𝘀 Same prompt ≠ same output. They predict tokens, not answers. You don’t pass parameters—you design prompts. That makes it hard to predict outputs, validate correctness, or reproduce behavior. 𝗣𝗿𝗼𝗺𝗽𝘁𝗶𝗻𝗴 𝗶𝘀𝗻’𝘁 𝗽𝗿𝗼𝗴𝗿𝗮𝗺𝗺𝗶𝗻𝗴 No function signatures. No modular reuse. Tiny prompt changes can break results. Long prompts increase latency. And prompts don’t always work the same across workflows or chains. 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗮𝗱𝗱𝘀 𝗺𝗼𝗿𝗲 𝘂𝗻𝗰𝗲𝗿𝘁𝗮𝗶𝗻𝘁𝘆 In RAG, you’re combining semantic search, reranking, and formatting. Each step adds noise. You’re generating over possibly irrelevant context. Now the system is doubly stochastic: retrieval + generation. 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗟𝗟𝗠𝘀 𝗶𝘀𝗻’𝘁 ���𝘁𝗿𝗮𝗶𝗴𝗵𝘁𝗳𝗼𝗿𝘄𝗮𝗿𝗱 There’s no .assertEqual(). Heuristic metrics are flawed. Human evals are expensive and inconsistent. Even stable outputs might still be wrong. 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗺𝗮𝗸𝗲𝘀 𝗶𝘁 𝗲𝘃𝗲𝗻 𝗵𝗮𝗿𝗱𝗲𝗿 LLMs are slow, expensive, and limited by token windows. You need chunking, caching, windowing, reranking, fallback logic. You’re not calling a model—you’re orchestrating a distributed system. 𝗗𝗲𝗯𝘂𝗴𝗴𝗶𝗻𝗴 𝗶𝘀 𝗮 𝗯𝗹𝗮𝗰𝗸 𝗯𝗼𝘅 No stack traces. No explanations. Logs give you input/output, not reasons. Prompt tweaks can cause side effects far from where you made the change. 𝗘𝘅𝗽𝗲𝗰𝘁𝗮𝘁𝗶𝗼𝗻𝘀 ≠ 𝗿𝗲𝗮𝗹𝗶𝘁𝘆 People expect memory, perfect instructions, stable outputs, and truth. LLMs forget, hallucinate, and drift based on sampling. Without scaffolding, they feel brittle and inconsistent. 𝗜𝗳 𝘆𝗼𝘂'𝗿𝗲 𝗹𝗼𝗼𝗸𝗶𝗻𝗴 𝗳𝗼𝗿 𝘃𝗶𝗯𝗲𝘀, 𝗹𝗲𝘁 𝗺𝗲 𝘁𝗲𝗹𝗹 𝘆𝗼𝘂 𝗶𝘁'𝘀 𝗼𝗻𝗹𝘆 𝗽𝗮𝗶𝗻 LLMs are probabilistic pattern matchers—not deterministic components. Building with them means thinking in systems, not functions. It means controlling chaos, not eliminating it. If your LLM system feels fragile, you’re not alone. You’re just facing reality. And it’s solvable—if you design for it. ♻️ Repost to share these insights. ➕ Follow Shivani Virdi for more.

On Sunday, I used AI to build an app that generated 23,000 lines of code in a couple of hours. On Monday, I posted about the experience, including the mistakes the AI made. My point: AI can build most of a tool fast, but you still need solid software engineering skills to ship a working product. Oddly, a bunch of software engineers jumped in to tell me I was using AI wrong. If I'd just followed better software engineering practices, they said, it would've worked perfectly. The irony? That was exactly my point. You need software engineering practices, knowledge and skills to get the most from AI. So for those that missed it, I've been documenting my views on AI-assisted coding. I believe in strong opinions, weakly held, so I'd like thoughts, feedback, and challenges to the following opinions. On AI and AI-Assisted Coding: → The agent harness largely shouldn’t matter. The process should work with all of them. → Most AI-assisted coding processes are too complex. They clutter the context window with unnecessary MCP tools, skills, or content from the AGENTS file. → A small, tightly defined, and focused context window produces the best results. → LLMs do not reason, they do not think, they are not intelligent. They're simple text prediction engines. Treat them that way. → LLMs are non-deterministic. That doesn't matter as long as the process provides deterministic feedback: compiler warnings as errors, linting, testing, and verifiable acceptance criteria. → Don't get attached to the code. Be prepared to revert changes and retry with refinements to the context. → Fast feedback helps. Provide a way for an LLM to get feedback on its work. → Coding standards and conventions remain useful. LLMs have been trained on code that follows common ones and to copy examples in their context. When your code align with those patterns, you get better results. On Software Development: → Work on small defined tasks. → Work with small batch sizes. → Do the simplest possible thing that meets the requirements. → Use TDD. → Make small atomic commits. → Work iteratively. → Refactor when needed. → Integrate continuously. → Trust, but verify. → Leverage tools. What are your strong opinions on AI-assisted coding?

LLMs will make us way worse at writing code by hand. If you think that’s a bad thing, you're not using them properly. I’ve conducted 250+ coding interviews at Uber and Airbnb over an 8 year period. We look for two main skills: 𝟭. 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 𝗦𝗼𝗹𝘃𝗶𝗻𝗴 - Can you gather requirements, iterate on ideas, find edge cases, design a solution? 𝟮. 𝗖𝗼𝗱𝗶𝗻𝗴 𝗔𝗯𝗶𝗹𝗶𝘁𝘆 - Can you translate that solution into clean code? LLMs are doing #2 not #1 Writing code is the 𝗹𝗮𝘀𝘁 𝘀𝘁𝗲𝗽 when you’re developing high quality software. You should research, plan and iterate on a design first. If you “vibe code” with prompts like “build me X”, thats jumping into code first, you’ll just ship bad software faster. If you use 𝗦𝗽𝗲𝗰-𝗗𝗿𝗶𝘃𝗲𝗻 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁, it’s different: • You research, plan, and iterate on ideas • You write a clear spec outlining the solution • The LLM turns that spec into code • You review and refine the result Solving math equations by hand was a valuable skill. Today it's practically useless and unnecessarily inefficient. The real skill is solving the science/engineering problem and as part of that identifying which calculations to perform. To even know which calculations to perform you still need to understand the equations, what they do and how to apply them. Same with LLMs and code. Even if you don’t hand-code every line, you still need to know what good code looks like so you can design it, guide it, and review it. You will get a lot of practice doing this by relying on LLMs for coding. If LLMs make us worse at implementing designs by hand, but force us to get better at analysis, design, and code review… That’s not the end of engineering. That’s the next version of it.