Challenges of AI in Software Development

🛠️ Can AI agents fix other #AI #Agents? A recent research paper — "Can Agents Fix Agent Issues?" — highlights this growing challenge in AI: how do we maintain LLM-powered agent systems as they become increasingly complex and mission-critical? The researchers analyzed 201 real-world issues across popular agent frameworks like MetaGPT and CrewAI, then built a reproducible benchmark (AGENTISSUE-BENCH) to test whether modern software engineering (SE) agents — such as SWE-agent, AutoCodeRover, and Agentless — can debug and fix these issues. And the results were interesting - even with top-tier models like GPT-4o and Claude 3.5 Sonnet, these SE agents could only correctly fix 3.33% to 12.67% of the problems. 👉 So why is it so hard to maintain AI agents than traditional software? I believe this could be because of LLM-based agents introducing a new dimensions of brittleness, i.e.: ◾ Prompt quality and structure impact behavior dramatically. ◾ Memory modules can silently corrupt state. ◾ External APIs/tools change without notice. ◾ LLM outputs are nondeterministic — the same prompt might behave differently each time. ◾ Workflows are dynamic and often fail in subtle, cascading ways. It simple words, maintaining AI agents is like managing a team of interns with varied expertise, who follow instructions vaguely, and have a tendency to improvise. Imagine a second team that monitors and corrects their behavior - without full visibility. Its a very tough ask right? 👉 Here are a few things that can be done to address this issue - to monitor and maintain agent systems at scale, we need a new layer of infrastructure — AgentOps - with capabilities like: ✅ Prompt trace logging: Track every prompt, model response, and tool invocation - like distributed tracing for microservices. ✅ Version-controlled memory: Treat agent memory like a database - with audit trails, rollback mechanisms, and schema validations. ✅ LLM-output validation: Use lightweight assertions or sanity checks to catch hallucinations or malformed output. ✅ Workflow watchdogs: Agents that observe other agents in real-time, detect hangs, infinite loops, or decision bottlenecks. ✅ Fine-grained test harnesses: Isolate agent actions and simulate edge cases before pushing to production workflows. ✅ Meta-agents: Purpose-built agents that debug, validate, and propose hotfixes using learnings from past failures. Bottom line: If LLM agents are the new application layer, we urgently need observability, fault tolerance, and debugging abstractions tailored to them. We're not just building smarter agents — we now need smarter agent infrastructure. #AI #LLM #AgentOps #AIInfrastructure #SoftwareEngineering #DevOps #Observability #AutonomousAgents #MLSystems #AIDebugging I write about #artificialintelligence | #technology | #startups | #mentoring | #leadership | #financialindependence   PS: All views are personal Vignesh Kumar

Several comprehensive studies including O’Reilly’s Playbook for Large Language Model Security, the 2025 State of Software Delivery report, and GitClear’s 2025 AI Copilot Code Quality report conclude that companies have started using AI for #coding too soon. A general conclusion: “LLMs are not #software engineers; they are like interns with goldfish memory. They’re great for quick tasks but terrible at keeping track of the big picture.” “As reliance on #AI increases, that big picture is being sidelined. Ironically, by certain accounts, the total developer workload is increasing—the majority of developers spend more time debugging AI-generated code and resolving security vulnerabilities.” “AI output is usually pretty good, but it’s still not quite reliable enough,” says another. “It needs to be a lot more accurate and consistent. Developers still always need to review, debug, and adjust it.” One problem: “AI tools tend to duplicate code, missing opportunities for code reuse and increasing the volume of code that must be maintained.” GitClear’s report “analyzed 211 million lines of code changes and found that in 2024, the frequency of duplicated code blocks increased eightfold.” “In addition to piling on unnecessary technical debt, cloned code blocks are linked to more defects—anywhere from 15% to 50% more.” While larger context windows will help, “they’re still insufficient to grasp full software architectures or suggest proper refactoring.” One CEO says: “AI tools often waste more time than they save for areas like generating entire programs or where broader context is required. The quality of the code generated drops significantly when they’re asked to write longer-form routines.” “Hallucinations still remain a concern. AI doesn’t just make mistakes—it makes them confidently. It will invent open-source packages that don’t exist, introduce subtle security vulnerabilities, and do it all with a straight face.” “Security vulnerabilities are another issue. AI-generated code may contain exploitable flaws.” Furthermore, AI agents often “fail to find root cause, resulting in partial or flawed solutions:” “Agents pinpoint the source of an issue remarkably quickly, using keyword searches across the whole repository to quickly locate the relevant file and functions—often far faster than a human would. However, they often exhibit a limited understanding of how the issue spans multiple components or files, and fail to address the root cause, leading to solutions that are incorrect or insufficiently comprehensive.” Solutions include better training data, more testing to validate AI outputs, progressive rollouts, and greater use of finely tuned models. The bottom line for some: “AI-generated code isn’t great—yet. But if you’re ignoring it, you’re already behind. The next 12 months are going to be a wild ride.” #technology #innovation #artificialintelligence #hype

AI Makes Software Supply Chain Attacks Even Worse 🧐 We've faced software supply chain attacks before, and in the AI era, these threats will only scale even further. It's crucial to rethink how we approach code and build security in this new reality. ⚠️ AI-driven coding tools are easy to use and productivity-boosting, but they're notoriously difficult to configure to align with organizational privacy and security policies. The genie is already out of the bottle, developers everywhere are adopting these tools rapidly. 🔙 Historical previous vulnerabilities get reintroduced: New AI-powered code generation trained on internal code repositories might unintentionally revive vulnerabilities previously patched. Why? Because LLMs prioritize functional correctness, not inherently secure code, and there's currently no robust, security-focused labeled dataset available to guide these models. The diversity of programming languages doesn’t make this problem any easier. 📉 Security reality check: The recent studies indicate that code generated by LLMs is only about ~40% secure even in optimal conditions. Functional correctness is not synonymous with security. 👉 https://baxbench.com 🤖⚡️ AI-agents already here, and they present a unique challenge: although they’re software, we often apply different (or insufficient) security standards or privacy policies. The risk of compromise or malicious takeover is real, and the consequences will intensify as these technologies will expose more to enterprises. New tech brings new responsibilities: I'm optimistic about AI’s long-term potential, but I’m deeply concerned about our readiness to defend against emerging threats at the pace AI adoption demands. The security guardrails we built just last year are already outdated and irrelevant in many cases. Tomorrow's threats require today's solutions. Traditional threat models and incident response playbooks no longer match AI-specific risks. We must proactively evolve our security mindset, practices, and tools to address the unique challenges of AI-era software development.

AI-driven software development is powerful, but it’s also full of traps. I keep seeing the same pitfalls repeat themselves, and most of them are easy to avoid if you know where to look. Almost every problem I’ve run into comes back to lack of specificity upstream. If your product spec leaves open questions, if you skip reviewing an implementation plan, or if you hand over ambiguity to the AI, it will show up later as costly complexity. You can absolutely use AI to weigh trade-offs, but you should never leave the decision-making to it at implementation time. Here are some of the most common pitfalls I’ve seen: 1) Implementing everything instead of choosing — if you leave multiple options open, the AI will often build all of them, add flags, or create branching logic instead of picking one path. 2) Gaming the tests — instead of solving the real problem, it may hardcode values, tweak the tests to match the code, or adjust the code to pass flawed tests. Either way, you end up with fragile, misaligned results. 3) Context blindness — the AI can miss system-level considerations like architecture, security, or performance constraints, producing code that works locally but adds hidden tech debt. 4) Code bloat — migration scripts, helpers, extra validation, “future-proofing” features…the AI happily generates more than you asked for, and rarely cleans up after itself. 5) Inconsistent conventions — code style, naming, and structure can drift from your team’s norms, leaving the codebase feeling fragmented. 6) Hidden dependencies — it may pull in libraries or packages you never intended, introducing maintenance or security issues. 7) Drift from intent — over time, it can wander away from the original product spec, slowly building toward something different than what you actually wanted. 8) Dead code — functions and logic used during debugging or earlier iterations often stick around, never removed. The good news is that these are all manageable. Review plans in detail. Eliminate ambiguity before handing things over. Keep product specs tight. Anchor the AI regularly to the original intent. And use tooling to catch dead code, dependency creep, and other artifacts. AI can absolutely accelerate development—but only if you stay disciplined in how you apply it.

𝐓𝐡𝐞 𝐀𝐈 𝐏𝐚𝐫𝐚𝐝𝐨𝐱 𝐢𝐧 𝐒𝐨𝐟𝐭𝐰𝐚𝐫𝐞 𝐃𝐞𝐯𝐞𝐥𝐨𝐩𝐦𝐞𝐧𝐭: 𝐌𝐨𝐫𝐞 𝐔𝐬𝐚𝐠𝐞, 𝐋𝐞𝐬𝐬 𝐂𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐜𝐞 The latest developer survey, featured by Ars Technica, is a wake-up call for technology leaders: trust in AI coding tools continues to fall, even as usage reaches all-time highs. Here’s what the numbers show—and what they mean for our engineering teams. 🎯𝐑𝐢𝐬𝐢𝐧𝐠 𝐔𝐬𝐚𝐠𝐞, 𝐄𝐫𝐨𝐝𝐢𝐧𝐠 𝐓𝐫𝐮𝐬𝐭 👉𝟖𝟒% of developers now use or plan to use AI coding tools, up from 76% last year—over half (51%) use them daily. ▪Yet trust in AI accuracy is at just 33%, and “high trust” is a mere 3%. Distrust is now at 46%, up sharply from 31% last year. ▪For experienced engineers, “high trust” plummets to just 2.6%. ▪Overall approval of AI is down from 72% in 2024 to only 60% this year. 🎯𝐑𝐞𝐚𝐥-𝐖𝐨𝐫𝐥𝐝 𝐈𝐦𝐩𝐚𝐜𝐭 👉𝟔𝟔% of developers say AI solutions are “almost right, but not quite”—leading to time-consuming bugs and more debugging, not less. ▪45% report debugging AI-generated code takes more time than expected. ▪29% believe AI tools struggle notably with complex tasks. ▪When the stakes are high, 75% would still turn to people for help rather than trust AI alone. ▪62% say ethical or security concerns require human judgment, not machine output. 🎯𝐖𝐡𝐚𝐭’𝐬 𝐍𝐞𝐱𝐭 𝐟𝐨𝐫 𝐋𝐞𝐚𝐝𝐞𝐫𝐬? 👉𝐄𝐦𝐛𝐞𝐝 𝐛𝐞𝐬𝐭 𝐩𝐫𝐚𝐜𝐭𝐢𝐜𝐞𝐬 ▪Human-in-the-loop review remains essential for mission-critical code and projects. 👉𝐅𝐨𝐬𝐭𝐞𝐫 𝐨𝐩𝐞𝐧 𝐝𝐢𝐚𝐥𝐨𝐠𝐮𝐞 ▪Encourage teams to surface frustrations—especially about “almost-right” AI outputs—and share solutions collaboratively. 👉𝐈𝐧𝐯𝐞𝐬𝐭 𝐢𝐧 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐞𝐫 𝐞𝐝𝐮𝐜𝐚𝐭𝐢𝐨𝐧 ▪Prioritize learning not just new AI-powered tools, but strategies for verification, auditing, and responsible adoption. The paradox is clear: AI is becoming more central in our workflows, not less. As adoption accelerates, let’s make sure trust and human expertise stay at the core of our development culture. 𝐒𝐨𝐮𝐫𝐜𝐞/𝐂𝐫𝐞𝐝𝐢𝐭: https://lnkd.in/grUheu9i #AI #DigitalTransformation #GenerativeAI #GenAI #Innovation  #ArtificialIntelligence #ML #ThoughtLeadership #NiteshRastogiInsights 

The Hidden Cost of AI: Why "Fast Coding" Might Be Expensive Coding 💰 Our recent analysis of AI-generated codebases reveals sobering truths about the hidden costs of "move fast and generate" approaches. The Numbers Don't Lie: 🟢 350% increase in technical debt accumulation rates with unreviewed AI code 🟢 Southwest Airlines' $390M meltdown partly attributed to legacy system technical debt 🟢 2020-2024 saw both 4x more code blocks AND 2x increase in code churn Y2K crisis-level maintenance challenges emerging in AI-heavy codebases Specific Technical Patterns We're Seeing: 🚨 Dependency Hell: AI tools generate code with antiquated dependencies, creating integration nightmares when systems need updates 🚨 Architecture Drift: Without proper review, AI-generated components bypass established patterns, creating inconsistent system architecture 🚨 Invisible Risk Accumulation: AI code often looks clean on the surface but contains subtle scalability issues that only emerge under load 🚨 Context Loss: AI generates solutions without understanding broader system implications—leading to tightly coupled, hard-to-maintain code Some Real-World Technical Debt Examples: ☑️ COBOL systems in banking (50+ years old) now interfacing with AI-generated Python microservices ☑️ Greenfield projects accumulating legacy debt within months due to inconsistent AI coding patterns ☑️ Critical infrastructure requiring complete rewrites after 18 months of AI-accelerated development What Leading Engineering Teams Are Doing: ✅ Architectural Guardrails: Pre-defined coding standards and design patterns that AI tools must follow ✅ Technical Debt Scoring: Automated tools that measure complexity, coupling, and maintainability of AI-generated code ✅ Hybrid Review Processes: Senior engineers reviewing AI output not just for bugs, but for long-term architectural impact ✅ Brownfield Strategy: Treating AI as a "greenfield vs brownfield" decision—different approaches for legacy integration vs new development A useful approach for organizations: teams should be experimenting with in-house AI training specifically focused on their existing codebase patterns—reducing architectural drift while maintaining velocity. Bottom Line for CTOs: The question isn't whether AI will create technical debt, but whether you're measuring and managing it proactively. Organizations treating AI as a "faster developer" rather than a "different kind of development approach" are setting themselves up for expensive surprises. The companies thriving with AI have learned that the real competitive advantage isn't just faster code—it's sustainable, maintainable code delivered faster. #SoftwareEngineering #AICodeGeneration #ai #EngineeringLeadership #TechDebt

In April, Microsoft said 30 percent of its code was written by AI. In December, Microsoft said most of Windows 11's core features are broken. There is no evidence the two are directly connected. Large codebases fail for many reasons, and correlation is not causation. But the sequence raises a question that the industry has been slow to address. AI adoption in software development is now measured in detail: percentage of code generated, developer hours saved, productivity metrics reported in earnings calls. These numbers are available because companies are proud of them. What remains unmeasured - at least publicly - is the long-term quality impact - Defect rates before and after AI adoption. - Maintenance burden on AI-generated code. - Rework cycles six months, twelve months, eighteen months out. Technical debt does not surface immediately. It compounds in the background and presents invoices later, often when the codebase has grown too complex to easily audit. Microsoft is unusually transparent. Most companies report the adoption curve without ever reporting the maintenance curve. Until both numbers exist, the real cost of AI-assisted development remains unknown. What long-term metric would change how you evaluate AI adoption in engineering teams? (Image: https://x.com/kmcnam1)

AI isn’t “free.” It comes with a hidden tax. I’m not talking about subscription fees or GPU costs. I’m talking about the real tax developers quietly pay every day when relying on AI tools: • Wasted time cleaning up half-correct code. • Losing muscle memory on fundamentals because autocomplete does the heavy lifting. • Chasing false confidence when an AI-generated solution “looks right” but fails in production. • Watching junior developers skip the struggle that actually teaches them why code works. AI feels like speed. But too often, it’s fake speed. Like a car that accelerates hard but has no steering. You get there faster… until you realize you’re headed straight into a wall. I’m not anti-AI. I use it daily. But we need to start talking about its hidden tax: the erosion of deep problem-solving skills and the mounting cleanup costs that come with trusting it too much. The question isn’t “Will AI take developer jobs?” The question is “What kind of developers will we become if we lean on it too hard?” #AIInDevelopment #HiddenCostsOfAI #DeveloperLife #CodeQuality #EngineeringExcellence #AIandDevelopers #FutureOfCoding #SoftwareCraftsmanship #CleanCode #DeveloperMindset #AIRealityCheck #ProductivityVsQuality #TechDebate #AIinSoftwareEngineering #DevelopersOnAI

“AI Debt” in coding is going to be one of the biggest challenges in the coming years. We’re only at the early stages, yet I’m already seeing orgs where a majority of the code is generated or modified by admins who have limited understanding of code structure, best practices, or established conventions. They experiment with AI-generated snippets, follow step-by-step LLM instructions, and feel the excitement when something finally works, but the long-term consequences often go unnoticed. Recently, I inherited an org where the custom code conflicted with a managed package, simply because it wasn’t designed or handled with proper development discipline. It worked “for now,” but it created a mess that was costly to clean up. AI tools can absolutely speed up delivery and make development more accessible, but without proper expertise and code review, they can also lead to massive technical debt, or what I call **“AI Debt.”** If you’re a business handing over your Salesforce org (or any system) to be built, please be cautious. Make sure the work is reviewed by someone who understands architecture, standards, and long-term sustainability. The short-term win isn’t worth the long-term pain. #Consulting #SalesforceConsulting #SalesforceAdmin #SalesforceDevelopment #SalesforceArchitect #AIDebt #SalesforceAI #AI