How to Address Mistakes in Software Development

Turns out "undefined" isn't a valid API key. Every 500 errors I’ve caused has taught me more than a successful deployment ever did. Backend engineering isn’t just about building systems. Sometimes, you break them, debug them, and learn. Here are 7 real mistakes that taught me more than any tutorial: 1. Forgot to set an environment variable: worked locally, blew up in prod. ✅ Don’t assume defaults exist ✅ Fail fast if critical configs are missing ✅ Validate env vars on startup—not after the app crashes 2. Didn’t handle a null or undefined field: classic edge case blind spot. ✅ Validate input and response data ✅ Use null-safe access patterns ✅ Add tests for edge cases 3. Relied on a 3rd-party API without a fallback: guess who had a bad day when it went down? ✅ Use retries with backoff ✅ Add fallback responses ✅ Gracefully degrade non-critical features 4. Improper timeout config: hello, hanging requests, and cascading failures. ✅ Set proper timeout values ✅ Handle timeout errors explicitly ✅ Monitor and tune under load 5. Race conditions in async code: everything’s fine... until it’s not under load. ✅ Avoid shared mutable state ✅ Use locks or atomic ops when needed ✅ Simulate load in test environments 6. Pushed a schema change without a data migration: and broke everything in 2 seconds. ✅ Pair schema changes with migrations ✅ Always test on staging with real data 7. Skipped input validation: the user sent a payload that wrecked my assumptions. ✅ Never trust client data ✅ Validate at the edge (API boundary) ✅ Enforce schemas and constraints You don’t become good by avoiding failure. You get there by surviving it. Failure isn’t a detour—it is the curriculum. Any lesson to add?

At some point in your career as a dev, you will break something. You may break it badly. You may make a terrible mistake, or have a bug that causes a real problem. Here’s some tips on handling it: 1) Take a breath. The stress of the realization may send your brain into fight-or-flight. In that mental state it is more difficult to think clearly and make good decisions. First, calm down. 2) Be honest. The problem may have or will trigger others’ stress as well. If you are calm and honest you will help them move out of problem panic into problem solving with you. Trust is built or lost in these moments. Coding mistakes aren’t the real source of lost trust. How we handle them is. 3) Be empathetic and determined. Acknowledge the negative impacts and feelings probably being felt by whoever is affected by the problem, and reassure them that you are determined to solve the problem. 4) Don’t treat it as a shocking event. Code will have bugs. Software is hard. Get others accustomed to this truth by not being shocked at it yourself. “Software has bugs but it will get fixed” is the reputation you want. “Their software never has bugs/problems” is an impossible reputation goal. 5) Get some breathing room, if you can. Ask yourself what the bug is preventing people from accomplishing, and determine if you can help them accomplish it another way, especially if it seems it will take some time to find or fix the problem. For example, once we had a web app crash when a mission-critical report was needed by our client. So we built the report manually from our DB and gave them a PDF. Bought us three months of breathing room until they needed the report again. 6) Focus on the problem, not the people. Keep morale up, including your own. Problems are harder to solve in a negative head space. Sometimes pressure can be useful, depending on the people, but harsh judgment does little to help. 7) Challenge your assumptions. Don’t get tunnel vision when it comes to the root of the problem. Consider every layer of your tech stack. Google around. Ask for help. Take a walk and clear your head. The answer is there, you just have to find it. If the issue is not a bug, but the result some other mistake you made, *still consider it a bug*. It’s a bug in the processes that should prevent human error from leaking into production. Work on fixing the results of the mistake, and then work on strengthening the process. —- You can get through a stressful dev-life moment. Just breathe!

In 2011, the Amazon Appstore failed on launch and Jeff Bezos was furious. It was my fault, and I handled one aspect of recovery so poorly that one of my engineers quit. I still regret it 14 years later. Please learn from my mistake. The main lesson is that when you are leading through a crisis, it can feel like it is all about you. It isn’t. It is about: 1) Solving the problem 2) Guiding your team through it The product issue was that there were some pretty simple bugs, and we solved those problem well enough that I was eventually promoted. Where I failed was in guiding my team through the crisis. My leadership miss was that I neglected to encourage and support the engineer who had written the bad code. He did a great job stepping up and supporting the effort to fix the problem, but shortly afterward, he resigned. During the crisis, I failed to make clear to him that we did not blame him for the launch failure despite the bugs. I imagine that left room for him to think we blamed him or that he didn’t belong. It is also possible that others did blame him directly and that I was too caught up in the crisis to realize it. Both instances were my responsibility as the leader of the team. His resignation taught me a valuable lesson about leading through a crisis: No matter how bad the situation is, your team must be your first priority. If you make them feel safe, they will move heaven and earth to fix the problem. If you don’t, they may still fix the problem, but the team itself will never be the same. As a leader, here is how you can give them what they need: 1) Take the blame and do not allow others to be blamed. In some bug cases after this we did not release the name of the engineer outside the team in order to protect them from judgment or blame. 2) Separate fixing the problem from figuring out why it happened. Once the problem is fixed, you can focus on root-causing. This lowers the risk of searching for answers getting confused with searching for someone to blame. 3) Realize that anyone involved in the problem already feels bad. High performers know when they have fallen short and let their team down. As a leader you have to show them the path to growth and success after the crisis. They do not need to be beaten up on- they have taken care of that themselves. 4) See crises and problems as growth opportunities, not personal flaws. Your team comes with you in a crisis whether you like it or not, so you might as well come out stronger on the other side. As a leader, the responsibility for a crisis is yours in two ways: The problem itself and the effect it has on the future of the team. Don’t get too caught up in the first to think about the second. Readers- Has your team survived a crisis? How did you handle it?

Have you ever spent endless hours on a project just to end up realising that a more straightforward method would have been more effective? This common mistake, referred to as over-engineering, can cause needless complexity and inefficiency when developing new products. Understanding Over-engineering > Over-engineering happens when a solution gets more difficult than it needs to be, usually by adding features or functionalities that do not directly meet the needs of customers. > This can lead to higher costs, longer development cycles, and less user-friendly products. Real-World Example: The Juicero The Juicero, a high-tech juicing machine, was released in 2016. It cost $700 and was designed to squeeze proprietary juice packets with considerable force. Later on, though, it was found that the costly machine was not essential because the same juice bags could be squeezed by hand. The company was eventually shut down as a result of the public outcry following this disclosure. My Own Story: The Overly Complex Website I was in a team early in my career that was assigned with creating a company website. We included the newest interactive elements and design trends in an effort to wow. Feedback received after the launch, however, indicated that visitors found the website overwhelming and challenging to use. In our pursuit of innovation, we had failed to realise the website's main purpose, which is to provide easily comprehensible information. I learnt the importance of simplicity and user-centred design from this experience. Useful Tips to Prevent Over-Engineering 1. Pay attention to the essential needs: Focus on key features that meet user needs and clearly explain the issue you're trying to solve. Don't include features that aren't directly useful. 2. Adopt Incremental Development: Begin with an MVP that satisfies the fundamental specifications. By using this method, you may get user input and decide on new features with knowledge. 3. Put Simplicity First: Use the KISS philosophy, which stands for "Keep It Simple, Stupid." Simpler designs are frequently easier to use and more efficient. 4. Verify Assumptions: Talk to users to learn about their wants and needs. This guarantees that the things you create will actually be useful to them. 5. Promote Open Communication: Create an environment where team members are at ease sharing thoughts and possible difficulties. Over-engineering tendencies can be recognised and avoided with the support of this collaborative environment. Have any of your initiatives involved over-engineering? How did you respond to it? Post your thoughts and experiences in the comments section below!

𝗪𝗵𝗮𝘁 𝗰𝗮𝗻 𝘄𝗲 𝗹𝗲𝗮𝗿𝗻 𝗳𝗿𝗼𝗺 𝘁𝗵𝗲 𝗕𝗼𝗲𝗶𝗻𝗴 𝟳𝟯𝟳 𝗠𝗔𝗫 𝗱𝗶𝘀𝗮𝘀𝘁𝗲𝗿? In October 2018 and March 2019, two Boeing 737 MAX aircraft crashed within five months of each other, claiming 346 lives. The problem was caused partially by a software system designed to enhance flight safety. MCAS Software was developed to address a fundamental design challenge in the 737 MAX. Due to its larger, more fuel-efficient engines, the MAX had different aerodynamic properties. MCAS was intended to make the MAX handle like earlier 737 models, ensuring a smooth transition for pilots. However, the system relied on input from just one of the plane's two angle of attack (AOA) sensors. If this single sensor provided incorrect data, MCAS could mistakenly push the plane's nose down repeatedly, overwhelming pilots who were often unaware of the system’s existence or behavior. Adding to the complexity, Boeing outsourced much of the 737 MAX's software development to engineers who were paid as little as $9 an hour, often recent graduates from overseas. What we can learn from the story as software engineers: 𝟭. 𝗥𝗲𝗺𝗼𝘃𝗲 𝗦𝗶𝗻𝗴𝗹𝗲 𝗣𝗼𝗶𝗻𝘁𝘀 𝗼𝗳 𝗙𝗮𝗶𝗹𝘂𝗿𝗲: The reliance on one AOA sensor created a critical vulnerability. In our work, we must ensure that no single component's failure can lead to catastrophic outcomes. 𝟮. 𝗞𝗲𝗲𝗽 𝗜𝘁 𝗦𝗶𝗺𝗽𝗹𝗲, 𝗕𝘂𝘁 𝗡𝗼𝘁 𝗧𝗼𝗼 𝗦𝗶𝗺𝗽𝗹𝗲 (𝗞𝗜𝗦𝗦): MCAS introduced new vulnerabilities as an overengineered fix to a hardware problem. We should prioritize simplicity without compromising safety. 𝟯. 𝗣𝗿𝗶𝗼𝗿𝗶𝘁𝗶𝘇𝗲 𝗗𝗼𝗺𝗮𝗶𝗻 𝗘𝘅𝗽𝗲𝗿𝘁𝗶𝘀𝗲: Outsourcing critical software to engineers without deep domain knowledge led to errors and miscommunication. Ensure that teams working on complex, safety-critical systems have the necessary expertise. 𝟰. 𝗦𝘁𝗿𝗲𝗻𝗴𝘁𝗵𝗲𝗻 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗣𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀: Despite extensive testing, the flaws in MCAS were not detected before the planes were deployed. This underscores the importance of robust testing, especially for difficult-to-replicate scenarios. What can we do daily as software engineers to improve these issues? Here are a few things: ✅ Implement proper error handling. Design your systems with the assumption that components will fail. Implement redundancies, cross-checks, and fail-safe mechanisms. ✅ Focus on users. Ensure that your systems communicate clearly with their users, especially about their current state and any automated actions. Provide users with the information and controls they need to make informed decisions and override automated systems when necessary. ✅ Prioritize integration testing. While unit tests are crucial, they're not enough. Invest time and resources in comprehensive, system-wide integration testing: test edge cases and failure modes. #technology #softwareengineering #programming #techworldwithmilan #coding

One of the easiest traps in software development is assuming success means everything is fine. A request can return a successful response and still hide serious issues underneath. What to look for with proper error handling: 🔹Don’t trust status codes alone — validate the data 🔹Handle failure states intentionally, not as an afterthought 🔹Log meaningful errors with enough context to debug later 🔹Surface useful messages to users while keeping internals secure 🔹Test what happens when things go wrong, not just when they go right Production doesn’t fail gracefully by default — engineers make it do that. The goal isn’t just working code. It’s resilient code that tells you what’s wrong before your users do. 💻

I wrote the perfect test case. Then, the bug hit production. And I said it... “Oops, I missed that bug.” But that moment made me ask a better question: What if we designed our systems to make mistakes more challenging to make in the first place? Enter: A brilliant (and wildly underrated) concept from Toyota’s production line— Poka-Yoke. 👀 What’s that? It means “mistake-proofing.” Not fixing bugs. Not catching them late. But stopping them before they ever happen. This blew my mind. And it’s not just for factories. It’s powerful for software, too. Here’s how Poka-Yoke shows up in testing: 🧩 Form Field Validations → Stop lousy input before it enters the system. ⚙️ Environment Pre-checks → Is the test environment right? The test doesn’t run. 🧹 Code Linters & Static Analysis → Catch issues before you ever hit “merge.” 🚫 CI/CD Pipeline Guards → Fail early if the code doesn’t meet the bar. 🖱️ Disable Buttons Until Fields Are Filled → A tiny UX tweak = huge bug savings. But here’s the real lesson: Poka-Yoke isn’t just a tactic. It’s a mindset shift. From reactive QA → to proactive quality engineering. 💬 Your turn— Where could a little mistake-proofing save you a massive headache in the future? #SoftwareTesting #QualityEngineering #Pokayoke #TestMetry