Building Test Suites for Code Verification

Before I let a language model write a single line of application code on a recent project, I spent two days building a test suite. Not a handful of unit tests. A conformance suite. Hundreds of examples harvested from open-source projects that solved adjacent problems. Their expectations, not their implementations. What does correctness look like, expressed as a contract? Then I added a benchmark. Not for optimisation — for detection. When the model refactors a function, does it still perform within acceptable bounds? Then generative tests. Random inputs, verified invariants. These find the cases you did not think of — and the model's failure modes are not the same as yours. Then mutation testing. Change a condition, flip a sign, delete a line. If the tests still pass, the tests are lying to you. Each guardrail has the same purpose: make quality observable. You cannot review every line of AI-generated code with the same attention you would give your own. But you can build a system that reviews it for you — not with understanding, but with rigour. The model is prolific. The guardrails are how you scale your judgment to match its output. Without them you are not engineering. You are hoping. Fourth in my series. Link in comments.

Most AI coders (Cursor, Claude Code, etc.) still skip the simplest path to reliable software: make the model fail first. Test-driven development turns an LLM into a self-correcting coder. Here’s the cycle I use with Claude (works for Gemini or o3 too): (1) Write failing tests – “generate unit tests for foo.py covering logged-out users; don’t touch implementation.” (2) Confirm the red bar – run the suite, watch it fail, commit the tests. (3) Iterate to green – instruct the coding model to “update foo.py until all tests pass. Tests stay frozen!” The AI agent then writes, runs, tweaks, and repeats. (4) Verify + commit – once the suite is green, push the code and open a PR with context-rich commit messages. Why this works: -> Tests act as a concrete target, slashing hallucinations -> Iterative feedback lets the coding agent self-correct instead of over-fitting a one-shot response -> You finish with executable specs, cleaner diffs, and auditable history I’ve cut debugging time in half since adopting this loop. If you’re agentic-coding without TDD, you’re leaving reliability and velocity on the table. This and a dozen more tips for developers building with AI in my latest AI Tidbits post https://lnkd.in/gTydCV9b

Don’t Focus Too Much On Writing More Tests Too Soon 📌 Prioritize Quality over Quantity - Make sure the tests you have (and this can even be just a single test) are useful, well-written and trustworthy. Make them part of your build pipeline. Make sure you know who needs to act when the test(s) should fail. Make sure you know who should write the next test. 📌 Test Coverage Analysis: Regularly assess the coverage of your tests to ensure they adequately exercise all parts of the codebase. Tools like code coverage analysis can help identify areas where additional testing is needed. 📌 Code Reviews for Tests: Just like code changes, tests should undergo thorough code reviews to ensure their quality and effectiveness. This helps catch any issues or oversights in the testing logic before they are integrated into the codebase. 📌 Parameterized and Data-Driven Tests: Incorporate parameterized and data-driven testing techniques to increase the versatility and comprehensiveness of your tests. This allows you to test a wider range of scenarios with minimal additional effort. 📌 Test Stability Monitoring: Monitor the stability of your tests over time to detect any flakiness or reliability issues. Continuous monitoring can help identify and address any recurring problems, ensuring the ongoing trustworthiness of your test suite. 📌 Test Environment Isolation: Ensure that tests are run in isolated environments to minimize interference from external factors. This helps maintain consistency and reliability in test results, regardless of changes in the development or deployment environment. 📌 Test Result Reporting: Implement robust reporting mechanisms for test results, including detailed logs and notifications. This enables quick identification and resolution of any failures, improving the responsiveness and reliability of the testing process. 📌 Regression Testing: Integrate regression testing into your workflow to detect unintended side effects of code changes. Automated regression tests help ensure that existing functionality remains intact as the codebase evolves, enhancing overall trust in the system. 📌 Periodic Review and Refinement: Regularly review and refine your testing strategy based on feedback and lessons learned from previous testing cycles. This iterative approach helps continually improve the effectiveness and trustworthiness of your testing process.

Here’s my step-by-step action plan whenever I work with a client to help them get a new automation project started. Maybe it’s useful to you, too. 0. Write a single, meaningful, efficient test. I don’t care if it’s a unit test, an integration test, an E2E test or whatever, as long as it is reliable, quick and produces information that is valuable. 1. Run that test a few times locally so you can reasonably assume that the test is reliable and repeatable. 2. Bring the test under version control. 3. Add the test to an existing pipeline or build a pipeline specifically for the execution of the test. Have it run on every commit or PR, or (not preferred) every night, depending on your collaboration strategy. 4. Trigger the pipeline a few times to make sure your test runs as reliably on the build agent as it does locally. 5. Improve the test code if and where needed. Run the test locally AND through the pipeline after every change you make to get feedback on the impact of your code change. This feedback loop should still be VERY short, as we’re still working with a single test (or a very small group of tests, at the most). 6. Consider adding a linter for your test code. This is an optional step, but one I do recommend. At some point, you’ll probably want to enforce a common coding style anyway, and introducing a linter early on is way less painful. Consider being pretty strict. Warnings are nice and gentle, but easy to ignore. Errors, not so much. 7. Only after you’ve completed all the previous steps you can start adding more tests. All these new tests will now be linted, put under version control and be run locally and on a build agent, because you made that part of the process early on, thereby setting yourself up for success in the long term. 8. Make refactoring and optimizing your test code part of the process. Practices like (A)TDD have this step built in for a reason. 9. Once you’ve added a few more tests, start running them in parallel. Again, you want to start doing this early on, because it’s much harder to introduce parallelisation after you’ve already written hundreds of tests. 10 - ∞ Rinse and repeat. Forget about ‘building a test automation framework’. That ‘framework’ will emerge pretty much by itself as long as you stick to the process I outlined here and don’t skip the continuous refactoring.

Let's talk about REAL API testing. Just ran our test suite: 88% line coverage, 59% branch coverage across 949 files. But that's not the flex - it's WHEN and HOW we run these tests that matters. Here's what most teams get wrong: They deploy to the dev environment first and then run API tests. The damage is already done. That's not testing; that's confirming bugs. Here's how we do it: 1. Developer raises PR 2. Spin up the infra within the pipeline like databases 3. Run the migrations 4. API tests run against the changes 5. Coverage reports show what we're missing 6. Code doesn't ship until tests pass Every. Single. Time. This gives us the confidence to ship weekly updates without breaking things. But here's the catch - don't worship coverage numbers blindly. A test with 100% coverage but weak assertions is worse than no test. It gives false confidence. We constantly evolve our test suite based on bug reports, adding assertions, and catching edge cases. Remember: Coverage tells you what code ran. Assertions tell you if it ran correctly. Ship with confidence, but make it earned confidence. #SoftwareTesting #QualityEngineering #SDET #BackendTesting #TestAutomation #SoftwareQuality

#𝐀𝐈-𝐀𝐬𝐬𝐢𝐬𝐭𝐞𝐝 ��𝐞𝐬𝐭 𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐨𝐧 | 𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐚 𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧-𝐋𝐢𝐤𝐞 𝐄2𝐄 𝐅𝐥𝐨𝐰 As part of my continuous learning in advanced QA, I recently designed and implemented a real-world end-to-end automation solution using Playwright (JavaScript). #𝗨𝘀𝗲 𝗖𝗮𝘀𝗲: Automating login flow for Booking.com using 𝗚𝗺𝗮𝗶𝗹-𝗯𝗮𝘀𝗲𝗱 𝗢𝗧𝗣 𝘃𝗲𝗿𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 — including capturing OTP directly from the email inbox. #𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 & 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗗𝗲𝘀𝗶𝗴𝗻: * Built a scalable and maintainable automation framework from scratch * Applied Page Object Model (POM) for clean separation of concerns * Created reusable utility classes (OTP handler, email parser, config manager) * Integrated environment management using dotenv * Focused on readability, reusability, and stability #𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗘2𝗘 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄: * Navigate to Booking.com * Handle dynamic UI elements (pop-ups, async loading) * Trigger Gmail authentication flow * Capture OTP from inbox in real-time * Parse and normalize OTP using regex-based logic #𝗥𝗲𝗮𝗹 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 (𝗣𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻-𝗟𝗲𝘃𝗲𝗹): * Gmail security & email authentication blocking * CA certificate issues in automation setup * Handling dynamic and flaky locators * Extracting correct OTP from unstructured email content #𝗦𝗼𝗹𝘂𝘁𝗶𝗼𝗻 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵: * Debugged failures with detailed logging & screenshots * Improved OTP extraction using multiple regex fallback strategies * Stabilized selectors using Playwright best practices * Used 𝗽𝗿𝗼𝗺𝗽𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝘄𝗶𝘁𝗵 𝗔𝗜 to accelerate debugging and code generation * Continuously validated outputs instead of blindly trusting AI #𝗜𝗺𝗽𝗮𝗰𝘁 & 𝗥𝗲𝘀𝘂𝗹𝘁: * Built a fully working E2E automation flow within ~2 hours * Successfully handled OTP-based authentication (a common real-world bottleneck) * Demonstrated fast prototyping + reliable validation mindset -AI is transforming how we build and test software. It can significantly speed up development, debugging, and idea generation. However, critical thinking, system understanding, and human validation remain irreplaceable for delivering production-quality solutions.Auto-fill verification code and complete login . #TestAutomation #Playwright #QA #E2ETesting #PromptEngineering #GmailAPI #OTPAutomation #RealWorldTesting

Are your tests actually testing anything? Or just performing "Mock Tautology Theatre"? This is where tests verify that your mocks behave exactly as you configured them. The test passes. The coverage report is green. But you've tested nothing real. The agent specifically hunts for this pattern — tests where the mock setup mirrors the implementation so closely that the "test" is just a tautology. I've extended my Test Design Reviewer agent for Claude Code with an emphasis on assessing the real value of tests in a project. My agent was already evaluating test suites against Dave Farley's 8 Properties of Good Tests and produces a quantitative Farley Index score (0-10). Here's the problem it solves: most teams measure test coverage but never measure test quality. You can have 95% coverage and still have a test suite that: • Tests mocks instead of behaviour (Mock Tautology Theatre) • Breaks every time you refactor (implementation coupling) • Passes even when production code is completely broken • Contains tests no one understands or maintains  What the agent evaluates (from Dave Farley's article):  8 properties, each scored independently: • Understandable — Do tests read like specifications? • Maintainable — Do they break only when they should? • Repeatable — Same result, every time, everywhere? • Atomic — Isolated, no side effects, parallelisable? • Necessary — Does every test earn its maintenance cost? • Granular — One behaviour per test, pinpoint diagnostics? • Fast — Millisecond feedback loops? • First (TDD) — Evidence of test-driven design?  How it works:  The agent uses a two-phase scoring methodology:  1. Static signal detection — deterministic pattern matching for sleep calls, reflection, shared state, cryptic names, mega-tests, trivial assertions  2. LLM semantic assessment — evaluating naming quality, assertion appropriateness, and design influence that static analysis misses These blend at 60/40 (static/LLM) to produce scores that are both reproducible and semantically aware. Every score is evidence-anchored — no score without a specific file:line reference. No hand-waving. The output: A structured report with the Farley Index, per-property breakdown (static + LLM + blended), a signal summary, the 5 worst-offending test methods, and prioritised improvement recommendations. Works across Python, Java, JavaScript/TypeScript, Go, and C#. Scales from small projects to large codebases via deterministic sampling. Built with the #nWave multi-agent framework for Claude Code to be released soon (stay tuned!) Feel free to use it on your projects and please give me feedback. All links in the first comment. #ClaudeCode #AI #ArtificialIntelligence #SoftwareDevelopment #SoftwareEngineering

How to start verifying a simple RTL design (Ex. Memory Controller) in UVM --> 1st Part: The very step is to understand the specification thoroughly with each of the below details: a. Working principle b. I/O ports and Signal level description c. Submodule details d. Is there any dependency of one signal with another, etc. e. Design detail understanding The next step is to build the Verification Plan and Feature / Testcase plan. List down the features that are needed to be verified. Some of them are below: ✓ Check whether clock and reset are working correctly. ✓ Check the correctness of the data by writing some value and read the same value of a single register. ✓ Repeat the process for all register. ✓ Check the default value of the register by reading the data. ✓ Check the controllers ability to handle multiple write and read transaction and Back to back writes. ✓ Check the correctness of WO register by reading from them. ✓ Check the correctness of RO register by writing on them. ✓ Check the Address translation and Address decoding issues being handled correctly. ✓ Check the invalid address access by RW from invalid or out of range addresses. ✓ Check the accessibility of locked condition of register. ✓ Check the controllers capability of accessing a single register from multiple address of different submap. ✓ Check the enablement and Quircky register accessibility. Above is a indicative list. Apart from the one mentioned above one also needs to add below cases: a. Stress testing. b. Use callbacks to inject errors. c. Multiple regression. d. Corner cases. The next step in order to verify the features verification planning is needed for which we need to code the following UVC's / objects along with Interface. a. Transaction b. Driver c. Monitor d. Sequencer e. Agent f. Sequence item g. Configuration h. Env i. Test #vlsi #asic #electronics #engineering

Contract testing can help you gain trust in your unit tests. Allowing you to create a lightning fast test suite that replaces your end to end tests. Here is how it works... Contract tests are tests for your test data suite. They work in conjunction with narrow integration tests. First you perform a narrow integration test against a real service, or database, and serialize the output of that test into a verification file. // load data from real service // serialize to file // compare serialized file to a previous run Secondly, you build up a test data suite with objects that correspond to objects in the output of the integration test. To proof they are equal, perform a contract test which serializes the test data and compares it against the verification file emitted by the integration test. // load data from test suite // serialize to file // compare serialized file to a previous run of integration test You have now proven that the test data is in conformance with the real data. And can now trust your test data suite, allowing you to reuse it in 1000s of unit and component tests without ever hitting the network. In combination with proper domain decomposition (according to the natural boundaries of the business capabilities), you can replace slow end to end tests with a series of: unit tests > contract tests > unit tests > contract tests > unit tests > contract test > ... This is possible because the output of one business capability is the input for the next, it serves as a contract in between the transitions. I would also advise to package the contract, the test data suite, the verification files and the contract tests into e.g. a nuget source code package, and share them between both capabilities (sometimes owned by different teams). This will allow both teams to run them and establish shared trust in the test suite. Your testing pyramid can now truly look something like this 1 Manual test 10 Integration tests 100 Contract tests 1000 Component tests 10000 Unit tests And your feedback loop will become super fast. I personally enforce a 10 second limit on the test runs, which will force the team to stick to less than 10 integration tests and do the rest of the testing in memory.

𝐀𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐧𝐠 - 𝐏𝐥𝐚𝐲𝐰𝐫𝐢𝐠𝐡𝐭 + 𝐏𝐎𝐌 𝐟𝐨𝐫 𝐚 𝐋𝐨𝐠𝐢𝐧 𝐏𝐚𝐠𝐞 When UI test suites grow, duplicate selectors + repeated steps quickly turn into flaky, hard-to-maintain tests. A simple fix is adopting the Page Object Model (POM): keep selectors and page actions in one place, and keep tests focused on behavior. In my Login Page example, I created a dedicated PracticeLoginPage page object that encapsulates: 𝐖𝐡𝐚𝐭 𝐭𝐡𝐞 𝐏𝐚𝐠𝐞 𝐎𝐛𝐣𝐞𝐜𝐭 𝐡𝐚𝐧𝐝𝐥𝐞𝐬? • Locators in one place (username, password, submit, success message, error message, logout) • A clean goto() + waitForPageLoad() to stabilize test setup • A reusable login(username, password) action for most scenarios • A “known good” loginWithValidCredentials() method using a static credential fixture • Optimized input helpers: → clear + fill (handles pre-filled values reliably) → character-by-character typing (useful for apps with real-time validation) 𝐓𝐞𝐬𝐭 𝐜𝐨𝐯𝐞𝐫𝐚𝐠𝐞 (𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐰𝐚𝐲𝐬 𝐭𝐨 𝐞𝐱𝐞𝐫𝐜𝐢𝐬𝐞 𝐭𝐡𝐞 𝐬𝐚𝐦𝐞 𝐩𝐚𝐠𝐞) Using beforeEach, each test starts with a fresh page object and navigates to the login screen. From there, the suite validates: • Successful login with valid credentials • Login via a single login() method (simple and readable) • Login via character-by-character typing (more realistic user simulation) • Invalid username → correct error message • Invalid password → correct error message • Input value verification after filling (asserts field state, not just outcomes) • Control state checks (username/password/submit are enabled) • End-to-end flow: login → assert success → logout 𝐖𝐡𝐲 𝐭𝐡𝐢𝐬 𝐩𝐚𝐭𝐭𝐞𝐫𝐧 𝐰𝐨𝐫𝐤𝐬 𝐰𝐞𝐥𝐥? • Tests stay short and intention-revealing (“what” not “how”) • Selectors change once (inside the page object), not across the suite • You can mix action styles (fill vs. typing) without rewriting tests • Built-in helpers (assertions/getters/state checks) make failures easier to diagnose If you’re building Playwright suites for real products, this is the type of structure that keeps your UI tests scalable, readable, and resilient as the app evolves. For a 1:1 connect 👉 https://lnkd.in/dEDZxwwF Comment "𝐅𝐫𝐚𝐦𝐞𝐰𝐨𝐫𝐤" to get the entire framework. #playwright #sdet #testautomation