Impact of Software Bugs on Code Reliability

One cost that's often missed, especially with older or legacy software is the one that comes from running with a very large backlog of unfixed bugs. I've worked with organisations who maintain software with 100s of open bugs, usually because the prioritisation process *only* looks at perceived business impact, and not at other factors such as operational impact or time to market. I see these bugs like barnacles on the hull of a boat, they are not very visible but significantly impact cost and efficiency, and eventually will cause serious long term damage. To give more details of a few of the ways "won't fix" bugs impact productivity: 1) Spending time and effort finding the bugs again and again with every release. One of the organisations I mentioned above burnt days of effort on each release (re-)finding bugs, and then cross referencing those against the known issues list before deciding, once again, not to fix them. 2) Workarounds in business processes and ways of working that are just there to deal with known issues with software. This impact is rarely ever tracked and soon becomes just business as usual. I've seen organisations rebuild these very same workarounds in brand new systems because they'd forgotten why they follow a particular business processes. (you can read more about this at https://lnkd.in/dTvpqgH) 3) Unfixed bugs make software more complex and much harder to understand, especially when workarounds start to creep in. In turn this makes software slower to change and increases the probability of yet more bugs creeping in. This is a feedback cycle that can lead to software having to be frozen or scraped. 4) Introduction of automated tests is more expensive and adds less value, either because they end up always broken (i.e. 100s of existing bugs) or add little value (lots of tests have to be disabled or ignored...). Time consuming manual regression testing often goes hand in hand with large bug backlogs. You really need a plan to fix large bug backlogs, and a prioritisation process that considers the total cost of a defect. Maybe start by committing to remove more bugs than you add on each release, add tests for bugs before you fix them to stop regressions, etc When I talk about "zero defect" approaches to software people cry "That's Impossible". It's not at all, especially when we start to look at the true costs of defects. In fact we could define bugs as being anything in our software that damages value, whether in the delivery of that software, in the business or for your customers.

Over the past week, the aviation world was shaken by news that thousands of Airbus A320-family aircraft needed emergency software fixes after a flight-control vulnerability was discovered. The trigger for the issue wasn’t “mysterious solar flares” or an unforeseeable cosmic event - it was something far more mundane and far more important for every industry that depends on software: a systemic failure to guard against data corruption in critical code paths. What actually happened? ✈️ All modern aircraft rely on fly-by-wire systems - computers interpret pilot inputs and translate them into actuator movements, taking into account myriads of sensor data and measurements. These systems are designed with layers of redundancy and error correction. But they still operate in the physical world, where high-altitude radiation can occasionally flip bits in memory or registers. Normally, defensive software detects and neutralizes these flips through validation, checksumming, and cross-channel comparison. In this case, a new version of the flight-control software called L104 - ironically deployed to increase system security - introduced a logic path that did not fully validate certain parameters after a bit-flip-type corruption. The failure mode was subtle: the software trusted data that should have been rejected. That allowed a rare but predictable environmental factor (radiation-induced bit errors) to propagate into control logic. This is the paradox every modern software engineering team faces: each new version is supposed to reduce risk, but when it reaches millions of lines of code and operates in safety-critical environments, even small validation gaps can create systemic vulnerabilities. In aviation, the consequences are dramatic. But the same pattern exists across cloud services, enterprise apps, OT systems, automotive, finance, and healthcare. The real lesson is not that one aircraft model had a bug - it’s that the scale and complexity of modern software have surpassed what manual processes can reliably assure. This is exactly why we’re building AISLE™: helping teams catch and remediate software vulnerabilities before they become fleet-wide recalls, security incidents, or outages. The Airbus episode is not an anomaly. It’s a preview of the world we now live in - one where resilience depends on our ability to verify and fortify software automatically, intelligently, and at scale. Building with Jaya Baloo, Stanislav Fort and team. Let’s go!! 🚀 #A320 #SoftwareSecurity #Resilience #AISLE

Inconsistent and intermittent failures tend to first reproduce long after the first time a check which exercises their code path is executed. This makes it almost impossible to know if a failure is from a new change, or from something that has been in the code for a long time. We had this problem in high volume in Microsoft Office due to an extremely large suite of end-to-end automated checks and a large number of shared tests. For years, the first place where intermittent failures would first appear would be when a developer ran their code assessing whether or not they had broken something prior to checking in. Roughly 85% of the time, developers after investigating the nature of the bug, determined it had nothing to do with their changes (I investigated this myself once - and I concurred, they were almost always correct in that assessment, and yes, that 85% is real). We flipped this trend by running all of the automation in the check-in suite many times per build. We would launch low-priority jobs on schedule against whatever the current build was consuming any unused capacity in the lab. Weekend builds typically got about 200 or so iterations, weekday between 50 and 100. The result was that most of the time by a wide margin, the first instance of an intermittent failure would come from these repeated runs. The system kept track so that when a developer saw the failure the report would indicate the issue was a known problem that had been in the code long before their changes were introduced. Meanwhile, these failures were tracked, investigated, and fixed mostly in order of frequency. We also had tooling that ran in the background trying different run and configuration parameters to try to increase frequency of hit and would notify product team engineers if it seemed like the bots were narrowing in on a repro condition. #softwaretesting #softwaredevelopment #embracetheflake

"How much is poor code quality actually costing us?" This question came up in a board meeting last month, and the engineering leader couldn't answer it. Here's the problem: Most companies treat code quality as an engineering concern, not a business concern. But poor code quality has a direct P&L impact: → Customer churn from bugs: When your app crashes, customers leave → Developer productivity loss: Engineers spending 40%+ of their time on technical debt → Opportunity cost: Features delayed because the codebase is too fragile to change quickly → Incident response costs: Engineers pulled off roadmap work to fight fires I just saw data from a company where improving their change failure rate by 15% translated to $2.3M in recovered engineering productivity over 12 months. Another company reduced their mean time to recovery and saw customer satisfaction scores improve by 18%, directly impacting renewal rates. The companies that get this right don't talk about code quality in technical terms. They talk about it in business terms: reduced risk, faster time-to-market, higher customer satisfaction. Because when you can quantify the ROI of quality, it stops being a "nice to have" and becomes a strategic business investment. How do you make the business case for code quality at your company? #EngineeringROI #TechnicalDebt #BusinessImpact #SoftwareQuality

How a few bits in the wrong place broke 1 billion computers and brought the world to standstill? 𝗧𝗵𝗲 𝗜𝗻𝗰𝗶𝗱𝗲𝗻𝘁 Recently, a serious issue emerged with Crowdstrike's software on Windows systems, causing widespread system crashes (Blue Screens of Death). This incident highlighted the critical nature of system driver stability and the potential far-reaching consequences of software errors at this level. 𝗥𝗼𝗼𝘁 𝗖𝗮𝘂𝘀𝗲 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 1. Null Pointer Exception: The core of the problem was a null pointer dereference, a common issue in memory-unsafe languages like C++. 2. Memory Access Violation: The software attempted to read from memory address 0x9c (156 in decimal), which is an invalid region for program access. Any attempt to read from this area triggers immediate termination by Windows. 3. Programmer Error: The issue stemmed from a failure to properly check for null pointers before accessing object members. In C++, address 0x0 is used to represent "null" or "nothing here." 4. System Driver Context: As the error occurred in a system driver with privileged access, Windows was forced to crash the entire system rather than just terminating a single program. 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗗𝗲𝘁𝗮𝗶𝗹𝘀 1. The error likely occurred when trying to access a member variable of a null object pointer. 2. The memory address being accessed (0x9c) suggests that the code was attempting to read from an offset of 156 bytes from a null pointer (0 + 0x9c = 0x9c). 3. This type of error is preventable with proper null checking or use of modern tooling that can detect such issues. 𝗜𝗺𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 𝗮𝗻𝗱 𝗙𝘂𝘁𝘂𝗿𝗲 𝗦𝘁𝗲𝗽𝘀 Microsoft's Role and Crowdstrike's Response • Need for improved policies to roll back defective drivers. • Potential enhancement of code safety measures. • Potential implementation of automated code sanitization tools. • Consideration of rewriting system driver in memory-safe language like Rust. • Industry-wide discussion on moving from C++ to safer languages like Rust. 𝗕𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗜𝗺𝗽𝗮𝗰𝘁 • Delivery services like FedEx, UPS, DHL face disruptions and delays. • Supermarkets struggle to accept mobile payments. • Major corporate IT worldwide struggles with point-of-purchase. • Major hospitals halt surgeries. • Airports ground and delay flights while engineers recover affected systems. • Repercussions spread to other platforms, with Amazon Web Services reporting issues. 𝗖𝗼𝗻𝗰𝗹𝘂𝘀𝗶𝗼𝗻 The losses incurred by businesses would easily be in 100 billions if not in trillions. This incident serves as a stark reminder of the importance of rigorous testing and safety checks in system-level software, especially in privileged contexts like drivers. It also highlights the ongoing challenges posed by memory-unsafe languages in critical software components. #crowdstrike #software #tech #microsoft #techtrend

Are minor bugs in your software spiraling into major setbacks? 😖 You might think it’s just part of the process, but it could be a sign of something bigger. Small defects can set off a chain reaction known as defect cascading, where one issue triggers another, then another, affecting the entire system. This isn’t just about individual errors, it’s about how interconnected components can turn a small glitch into widespread failures that jeopardize entire projects. These cascading defects disrupt performance and can significantly damage client trust when deliverables don’t meet expectations. The challenge lies in detecting these issues early before they escalate and become more complex and costly to resolve. Proactive quality assurance measures are essential. Teams can better manage these risks by integrating comprehensive testing strategies and maintaining rigorous standards throughout the development cycle. This approach not only prevents the initial defects but also stops them from triggering further issues down the line. Strengthening your software development practices with robust defect management techniques ensures your projects remain on track, perform reliably, and maintain the high quality your clients expect. Isn’t it time to take control of your software’s quality and prevent these hidden pitfalls from undermining your projects? 🚀 #SoftwareDevelopment #DevOps #DefectManagement

Could the colossal failure of CrowdStrike's Falcon platform that took down over 8 million Windows servers and caused severe damage to computer systems around the world have been prevented with basic software engineering practices including a better design, good QA and staged roll outs? Poor Design The bug was caused by a threat config file update in CrowdStrike’s Falcon platform that would cause the Falcon software to access invalid memory. The config file is often updated to make the platform aware of new security threats - sometimes it’s updated multiple times a day. Since the Falcon software runs at the OS level as a driver, the invalid memory access would cause the machine to crash and reboot. Upon reboot normally the bad driver code would be ignored, but not in this case - the Falcon software was deemed a ‘Critical Driver’ and so upon each restart, the OS would try to run it again. Preventing hacker malware is important - but is it more important than running the system at all? Instead a more robust design would allow for the system to reboot safely without CrowdStrike in the mix or at a minimum, give admins the ability to configure this as an option remotely. Microsoft's Windows weak OS design, which has been an issue for years, is at the core of this issue. It is important to note that many other machines Linux, Apple and other platforms were unaffected. In this particular case this update was specific to the Microsoft OS. In other cases however, Unix based OS have a better design in place that would protect it from the catastrophic doom loop that effected Microsoft Windows machines. Lack of basic QA Often software bugs can be very tricky only appearing in certain rare edge cases. These types of bugs are very difficult to catch and test. This was not the case. In this case, with over 8 M computers effected, it seemed to be happening every time. Some basic QA process should have caught this. A Staged Rollout Strategy would have been a better strategy When deploying critical systems, it's wise to release new code in a smaller, controlled environment first. This helps identify and fix any bugs that could cause catastrophic consequences before a wider rollout. This is a lesson we learned early on at ICE. We adapted by rolling out major updates to certain smaller markets first before rolling it out to the systems that managed Global Oil trading for example. If there was a bug in the initial rollout we could pause, fix it and then try again and the other markets would never have been affected. 

𝐀𝐈 𝐖𝐫𝐢𝐭𝐞𝐬 𝐌𝐨𝐫𝐞 𝐂𝐨𝐝𝐞 – 𝐀𝐧𝐝 𝐌𝐨𝐫𝐞 𝐁𝐮𝐠𝐬: 𝐖𝐡𝐚𝐭 𝐭𝐡𝐞 𝐃𝐚𝐭𝐚 𝐀𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐒𝐡𝐨𝐰𝐬 AI-generated code is accelerating software delivery but also shipping significantly more defects than human-written code, especially around logic, security, and performance. It shifts developer focus from typing code to reviewing, testing, and governing AI output. As teams rush to adopt AI coding assistants, a new #CodeRabbit report highlights a clear trade-off: more code and faster drafts, but also more issues, deeper security risks, and heavier review loads. 🔹𝐊𝐞𝐲 𝐟𝐢𝐧𝐝𝐢𝐧𝐠𝐬 👉 𝐈𝐬𝐬𝐮𝐞 𝐯𝐨𝐥𝐮𝐦𝐞 ▪AI-generated pull requests average 10.83 issues vs 6.45 for human PRs (around 1.7x more). ▪AI-authored PRs also include 1.4x more critical issues and 1.7x more major issues. 👉 𝐃𝐞𝐟𝐞𝐜𝐭 𝐜𝐚𝐭𝐞𝐠𝐨𝐫𝐢𝐞𝐬 ▪Logic and correctness errors appear about 1.75x more often in AI-generated code. ▪Code quality and maintainability issues are 1.64x higher, with readability problems increasing more than 3x in some analyses. 👉 𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲 𝐫𝐢𝐬𝐤𝐬 ▪Security vulnerabilities rise roughly 1.5–1.57x in AI-generated code. ▪Common issues include improper password handling, insecure object references, XSS vulnerabilities, insecure deserialization. 👉 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐚𝐧𝐝 𝐫𝐞𝐥𝐢𝐚𝐛𝐢𝐥𝐢𝐭𝐲 ▪Performance-related issues are around 1.42x more common, including inefficient I/O and suboptimal resource usage. ▪These issues lengthen reviews and increase the chance that serious bugs slip into production. 👉 𝐖𝐡𝐞𝐫𝐞 𝐀𝐈 𝐡𝐞𝐥𝐩𝐬 ▪AI-generated code shows 1.76x fewer spelling errors and 1.32x fewer testability issues, improving surface-level polish. ▪AI dramatically increases output volume, shifting human effort toward review, risk assessment, and higher-order design. 🔹𝐓𝐚𝐤𝐞𝐚𝐰𝐚𝐲𝐬 ▪Treat AI as a force multiplier, not an autopilot: pair AI coding tools with strong code review culture, threat modeling, and CI/CD gates. ▪Invest in governance: enforce linters, formatters, security scanners, and explicit AI usage policies to catch AI-specific failure modes early. ▪Upskill teams: train developers to recognize typical AI mistakes in logic, security, and performance, and to design prompts that incorporate business rules and architectural constraints. AI coding tools are here to stay, but this research is a reminder that speed without guardrails quickly turns into risk. The competitive advantage will belong to teams that combine AI-assisted generation with disciplined practices, rigorous review, a security-first mindset from day one. 𝐒𝐨𝐮𝐫𝐜𝐞/𝐂𝐫𝐞𝐝𝐢𝐭: https://lnkd.in/g9ctpXDf https://lnkd.in/g7AUt2Kq #AI #AgenticAI #DigitalTransformation #GenerativeAI #GenAI #Innovation  #ArtificialIntelligence #ML #ThoughtLeadership #NiteshRastogiInsights  ---------------------------------------------------------------------- • Please 𝐋𝐢𝐤𝐞, 𝐒𝐡𝐚𝐫𝐞, 𝐂𝐨𝐦𝐦𝐞𝐧𝐭, 𝐒𝐚𝐯𝐞, 𝐅𝐨𝐥𝐥𝐨𝐰 https://lnkd.in/gUeJrb63