Software Testing Basics

Demystifying the Software Testing 1️⃣ 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝗧𝗲𝘀𝘁𝗶𝗻𝗴: 𝗧𝗵𝗲 𝗕𝗮𝘀𝗶𝗰𝘀: Unit Testing: Isolating individual code units to ensure they work as expected. Think of it as testing each brick before building a wall. Integration Testing: Verifying how different modules work together. Imagine testing how the bricks fit into the wall. System Testing: Putting it all together, ensuring the entire system functions as designed. Now, test the whole building for stability and functionality. Acceptance Testing: The final hurdle! Here, users or stakeholders confirm the software meets their needs. Think of it as the grand opening ceremony for your building. 2️⃣ 𝗡𝗼𝗻-𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝗧𝗲𝘀𝘁𝗶𝗻𝗴: 𝗕𝗲𝘆𝗼𝗻𝗱 𝘁𝗵𝗲 𝗕𝗮𝘀𝗶𝗰𝘀: ️ Performance Testing: Assessing speed, responsiveness, and scalability under different loads. Imagine testing how many people your building can safely accommodate. Security Testing: Identifying and mitigating vulnerabilities to protect against cyberattacks. Think of it as installing security systems and testing their effectiveness. Usability Testing: Evaluating how easy and intuitive the software is to use. Imagine testing how user-friendly your building is for navigation and accessibility. 3️⃣ 𝗢𝘁𝗵𝗲𝗿 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗔𝘃𝗲𝗻𝘂𝗲𝘀: 𝗧𝗵𝗲 𝗦𝗽𝗲𝗰𝗶𝗮𝗹𝗶𝘇𝗲𝗱 𝗖𝗿𝗲𝘄: Regression Testing: Ensuring new changes haven't broken existing functionality. Imagine checking your building for cracks after renovations. Smoke Testing: A quick sanity check to ensure basic functionality before further testing. Think of turning on the lights and checking for basic systems functionality before a deeper inspection. Exploratory Testing: Unstructured, creative testing to uncover unexpected issues. Imagine a detective searching for hidden clues in your building. Have I overlooked anything? Please share your thoughts—your insights are priceless to me.

The AI Coding Revolution Is Here, But Are We Testing for It? As AI-assisted development reshapes how we build software, I've been thinking a lot about something that is talked about often but doesn't always get the focus it deserves: automated testing. At JPMorganChase, we're embracing AI coding tools to accelerate delivery, reduce toil, and empower our teams to focus on the work that matters, reducing cognitive load of repetitive tasks. But speed without safety is just risk in disguise. Here's what I believe every leader (and this is broader than technology) needs to consider right now: �� AI writes code faster than humans can review it manually. If your testing strategy is still largely manual, you're already behind. AI-generated code can introduce subtle logic errors, security vulnerabilities, or edge-case failures that look perfectly reasonable on the surface. Automated testing is no longer a best practice, it's a non-negotiable safeguard. • Test coverage is your new quality contract. When AI is your co-developer, the test suite becomes the specification. If you can't describe expected behavior in a test, you can't trust what the AI builds. Investing in robust unit, integration, and regression testing frameworks is investing in the integrity of your entire delivery pipeline. • Shift-left testing amplifies AI's value. It doesn't slow it down. Some worry that rigorous testing will negate the speed gains from AI coding. The opposite is true. When automated tests are embedded early in the development lifecycle, AI tools can iterate faster, self-correct, and validate outputs in real time. Testing enables velocity; it doesn't constrain it. • Your teams need to evolve alongside the tools. The best teams of tomorrow won't just write code. They'll architect test strategies, evaluate AI outputs critically, and build systems that are observable and verifiable by design. We owe it to our teams to invest in this skill evolution now. At the scale we operate, serving millions of customers, the cost of a defect isn't just technical. It's trust. And trust, once broken, is hard to rebuild. AI is a force multiplier. But multiplying without a strong foundation multiplies risk just as fast as it multiplies output. Build fast. Test smarter. Ship with confidence. I'd love to hear how other leaders are thinking about quality engineering in the age of AI. What's working for your teams? #AIEngineering #SoftwareTesting

𝗧𝘆𝗽𝗲𝘀 𝗼𝗳 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗧𝗲𝘀𝘁𝗶𝗻𝗴: 𝗔 𝗖𝗼𝗺𝗽𝗿𝗲𝗵𝗲𝗻𝘀𝗶𝘃𝗲 𝗢𝘃𝗲𝗿𝘃𝗶𝗲𝘄 𝟭. 𝗠𝗮𝗻𝘂𝗮𝗹 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 Manual testing involves human effort to identify bugs and ensure the software meets requirements. It includes: 𝐖𝐡𝐢𝐭𝐞 𝐁𝐨𝐱 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Focuses on the internal structure and logic of the code. 𝐁𝐥𝐚𝐜𝐤 𝐁𝐨𝐱 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Concentrates on the functionality without knowledge of the internal code. 𝐆𝐫𝐞𝐲 𝐁𝐨𝐱 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Combines both White Box and Black Box techniques, giving partial insight into the code. 𝟮. 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 Automation testing uses scripts and tools to execute tests efficiently, ensuring faster results for repetitive tasks. This approach complements manual testing by reducing time and effort. 𝟯. 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 Functional testing verifies that the application behaves as expected and satisfies functional requirements. Subtypes include: 𝐔𝐧𝐢𝐭 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Validates individual components or units of the application. 𝐔𝐬𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Ensures the application is user-friendly and intuitive. 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝘁𝗲𝘀𝘁𝗶𝗻𝗴 𝗳𝘂𝗿𝘁𝗵𝗲𝗿 𝗲𝘅𝘁𝗲𝗻𝗱𝘀 𝘁𝗼 :- 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Tests the interaction between integrated modules. It has two methods: 𝗜𝗻𝗰𝗿𝗲𝗺𝗲𝗻𝘁𝗮𝗹 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 :- 𝐁𝐨𝐭𝐭𝐨𝐦-𝐔𝐩 𝐀𝐩𝐩𝐫𝐨𝐚𝐜𝐡: Starts testing with lower-level modules. 𝐓𝐨𝐩-𝐃𝐨𝐰𝐧 𝐀𝐩𝐩𝐫𝐨𝐚𝐜𝐡: Begins testing with higher-level modules. 𝐍𝐨𝐧-𝐈𝐧𝐜𝐫𝐞𝐦𝐞𝐧𝐭𝐚𝐥 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Tests all modules as a single unit. 𝐒𝐲𝐬𝐭𝐞𝐦 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Tests the entire system as a whole to ensure it meets specified requirements. 𝟰. 𝗡𝗼𝗻-𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝗮𝗹 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 Non-functional testing evaluates the performance, reliability, scalability, and other non-functional aspects of the application. Key subtypes include: 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 :- 𝐋𝐨𝐚𝐝 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Checks the application's behavior under expected load. 𝐒𝐭𝐫𝐞𝐬𝐬 𝐓𝐞𝐬𝐭𝐢𝐧𝐠:Tests the application's stability under extreme conditions. 𝐒𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Assesses the application's ability to scale up. 𝐒𝐭𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐓𝐞𝐬𝐭𝐢𝐧𝐠:Ensures consistent performance over time. 𝐂𝐨𝐦𝐩𝐚𝐭𝐢𝐛𝐢𝐥𝐢𝐭𝐲 𝐓𝐞𝐬𝐭𝐢𝐧𝐠: Verifies that the application works across various devices, platforms, or operating systems. 𝗪𝗵𝘆 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗠𝗮𝘁𝘁𝗲𝗿𝘀 Testing ensures a bug-free, reliable, and high-performing application. By combining manual and automated approaches with functional and non-functional testing techniques, developers can deliver a robust product that meets both user expectations and business requirements. Understanding these testing types helps teams choose the right strategy to achieve software excellence!

How to Write UAT Test Cases (for Business Analysts) I remember when I was a junior BA - the idea of UAT scared me. I didn’t want to admit I didn’t know where to start... So I stayed quiet and tried to figure it out on my own. Turns out, I’d built it up to be more complicated than it really is. Here’s what I learned: As a BA, your role in User Acceptance Testing (UAT) is to ensure the solution actually meets the business need… not just that it functions. To do that effectively, you need a structured approach to writing UAT test cases. Here's how I do it: 1️⃣ Start with the Requirement → Begin with a single requirement or user story. → Each requirement must be tested, and depending on how many acceptance criteria it has, you may need multiple test cases. (Think: What is the business expecting from this requirement?) 2️⃣ Review the Acceptance Criteria → Acceptance criteria define the boundary of success for a requirement. → They help you understand what “good” looks like from the business’s perspective. (Use these criteria as your guideposts for what to test) 3️⃣ Develop Test Cases Based on the Acceptance Criteria → Each acceptance criterion should translate into at least one test case. → Some may need both a positive (happy path) and negative (error or edge case) scenario. (If a criterion says “User must receive a confirmation email,” test both a valid scenario and one where the email fails) 4️⃣ Complete the UAT Template for Each Test Case → For each test case, fill in these fields: ☑ Test Description – A clear statement of what’s being tested e.g. “Test password reset email is triggered for valid email addresses” ☑ Preconditions – Any setup required before testing e.g. “User is logged out and on the login page” ☑ Test Steps – Step-by-step actions for the tester to perform e.g. Click “Forgot Password”, enter email, submit form ☑ Expected Result – What should happen if the system works correctly e.g. “User receives reset email within 2 minutes” (TIP: Keep the language business-friendly so anyone can run the test) 5️⃣ Repeat for Each Requirement → Once you've completed the test cases for one requirement, move to the next and repeat the process. → This ensures full coverage and traceability back to each business objective. 6️⃣ Review with Business Stakeholders → Once your test cases are drafted, share them with your business SMEs or stakeholders. (This step is critical - their feedback confirms that you’re testing what really matters to them) 7️⃣ Prepare for Execution → After validation, the test cases are ready to be run. → Depending on your project, UAT may be carried out by business users, or you may help execute or facilitate it as a BA. 📩 Want a copy of my UAT test case template? → Send me a message and I’ll be happy to share it with you 😊 Found this interesting? Repost to your network, and follow me → Matthew Thomas Holliday #BusinessAnalysis #UAT #BAskills #BAmethods #UATtemplate

Why Every Product Manager Needs A/B Testing 🚀 Imagine cooking up a recipe for the perfect product feature. Would you trust your instincts blindly, or would you test different ingredients to get the best taste? That’s where A/B testing comes in. It’s the secret sauce that helps Product Managers make data-driven decisions with confidence. Here’s everything you need to know to master A/B testing: ❓ What is A/B Testing❓ A/B testing is the process of comparing two or more versions of a product to determine which one performs better. The versions might differ in small ways - a new button design, a revamped landing page, or an updated pricing structure but the impact on user behaviour can be monumental. This method helps you validate assumptions, optimize user experiences, and ensure every product decision adds value. ⚙️ How to Conduct a Successful A/B Test? ⚙️ 🔹 Set Clear Goals Ask yourself what are you trying to improve? It could be anything from conversion rates to user satisfaction. Your goal is your North Star. 🔹 Choose the Right Metrics Metrics like click-through rates (CTR), time spent on a page, or purchase frequency will guide you in evaluating success. 🔹 Hypothesize Frame your test with a simple prediction. Example: “I believe changing the CTA button color from blue to green will increase clicks by 15%.” 🔹 Design Your Experiment Define your control group (current version) and treatment group (variant to test), ensuring a large enough sample size for reliable results. Run the test for a sufficient duration to capture meaningful patterns and user behaviour. 🔹 Analyze & Implement Use tools like Google Optimize or Optimizely to analyze results and determine statistical significance. Roll out the winning variant confidently, or refine your hypothesis for future iterations if results are inconclusive. ♻️ Four Types of A/B Tests Every PM Should Know ♻️ 1️⃣ Feature Testing: Validate hypotheses for new features pre-launch. 2️⃣ Live Testing: Fine-tune existing features already in the wild. 3️⃣ Trapdoor Testing: Redirect traffic between variants dynamically. 4️⃣ Multi-Armed Bandit (MAB): Let machine learning allocate traffic to better-performing variants in real-time. ❌ Common Pitfalls to Avoid ❌ 1️⃣ Testing trivial changes that won’t move the needle. 2️⃣ Ignoring sample size requirements—small audiences lead to inaccurate conclusions. 3️⃣ Treating A/B testing as a one-off exercise. Optimization is an ongoing journey. What’s been your most surprising A/B testing discovery? Let’s discuss in the comments!👇 Ready to embark on an exhilarating journey into the heart of product management? I’ve recently launched a cohort that is focused on teaching end-to-end product management as well as providing career placement opportunities! 🧠 Fill in the form in the comments to register your interest in the cohort and I’ll reach out to you with further details. ✍️ #ProductManagement #ABTesting #PMTools #ContinuousOptimization

A/B testing is essential for companies that want to make product decisions based on real user data. It helps teams move beyond gut feeling and build better experiences over time. In a recent blog post, the Data Science team at Oda shared their journey of building a strong online experimentation culture. Their journey unfolded through three main phases. In the early phase, tests were run manually by engineers. As the company grew, they introduced a centralized experimentation platform. Eventually, they reached a mature phase where experimentation was democratized across teams. Along the way, they discovered that building strong organizational processes was just as important as developing the technical platform. They stressed the importance of clear documentation, good experiment design practices, and fostering a mindset that treats failed experiments as valuable learning opportunities. Their experience highlights that success in A/B testing comes from tools and creating an environment where experimentation is trusted and encouraged. The blog offers valuable insights into the phases of A/B testing and the lessons Oda learned — and serves as a great reminder that building a thriving experimentation culture requires both technical investment and organizational commitment. #DataScience #Experimentation #ABTesting #Analytics #Culture #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gXb68hSb 

🚀 QA vs QC vs QE vs Tester: Do they really mean the same thing? In the world of software development, we often hear these terms as if they were interchangeable... but they are not. Each role adds value from a different perspective within quality. Here is a brief and practical 👇 explanation 🔵 QA – Quality Assurance ➡️ Prevents defects ➡️ Define processes, standards and best practices ➡️ Focuses on how software is built ➡️ More strategic and preventive work Think of QA as, "Let's get it right from the start." 🟣 QC – Quality Control ➡️ Detect defects ➡️ Review the finished product or in specific phases ➡️ Focuses on what is delivered ➡️ More reactive work, evaluating the final result QC is: "Let's validate that what was done meets expectations." 🟢 QE – Quality Engineering ➡️ Unites quality + engineering ➡️ Automation, CI/CD, Quality Metrics, Tools ➡️ Ensures quality through technology and engineering ➡️ Key role in DevOps/Agile teams QE is, "Let's make quality flow into the pipeline." 🟡 Tester – Test Analyst/Engineer ➡️ Run manual or automated tests ➡️ Understand requirements, design test cases, report bugs ➡️ Ensures software works as expected ➡️ Direct focus on product behavior Tester is: "Let's prove that everything works... and find what we don't." 💬 Final Thoughts Quality is not the responsibility of a single person, but these roles help make it a natural part of the development cycle. Understanding the differences allows us to collaborate better and build more reliable software.

founder learnings! part 8. A/B test math interpretation - I love stuff like this: Two members of our team (Fletcher Ehlers and Marie-Louise Brunet) - ran a test recently that decreased click-through rate (CTR) by over 10% - they added a warning telling users they’d need to log in if they clicked. However - instead of hurting conversions like you’d think, it actually increased them. As in - Fewer users clicked through, but overall, more users ended up finishing the flow. Why? Selection bias & signal vs. noise. By adding friction, we filtered out low-intent users—those who would have clicked but bounced at the next step. The ones who still clicked knew what they were getting into, making them far more likely to convert. Fewer clicks, but higher quality clicks. Here's a visual representation of the A/B test results. You can see how the click-through rate (CTR) dropped after adding friction (fewer clicks), but the total number of conversions increased. This highlights the power of understanding selection bias—removing low-intent users improved the quality of clicks, leading to better overall results.

𝗜𝘁 𝘄𝗼𝗿𝗸𝗲𝗱 𝗶𝗻 𝘁𝗲𝘀𝘁𝗶𝗻𝗴. 𝗨𝗻𝘁𝗶𝗹 𝗶𝘁 𝗱𝗶𝗱𝗻’𝘁. The model passed every unit test. The UI felt smooth. The demo? Flawless. Then users showed up. And suddenly... → Retrieval returned the wrong context → The model started responding like it had amnesia → Edge-case prompts broke the flow → Confidence quietly slipped No big crashes. No obvious bugs. Just a slow unraveling of trust. Most AI products don’t fail because they’re broken. They fail because they weren’t tested the way real people actually use them. And that’s the trap. You test clean. They use messy. That’s why we built RagMetrics — to help AI teams test beyond the “happy path.” ✅ Real-world prompts ✅ Retrieval stress tests ✅ Output consistency checks ✅ Actual behavior under pressure — not assumptions Because fixing it after launch is expensive. But losing trust? That’s even harder to repair. If you’ve ever shipped something that “worked”… until a real user touched it — you know exactly what I’m talking about. Let’s not find out the hard way. 🧠 How are you testing your LLM in the wild? Would love to hear what’s working for you.