Improving Prototype Code with Automated Testing

Here’s my step-by-step action plan whenever I work with a client to help them get a new automation project started. Maybe it’s useful to you, too. 0. Write a single, meaningful, efficient test. I don’t care if it’s a unit test, an integration test, an E2E test or whatever, as long as it is reliable, quick and produces information that is valuable. 1. Run that test a few times locally so you can reasonably assume that the test is reliable and repeatable. 2. Bring the test under version control. 3. Add the test to an existing pipeline or build a pipeline specifically for the execution of the test. Have it run on every commit or PR, or (not preferred) every night, depending on your collaboration strategy. 4. Trigger the pipeline a few times to make sure your test runs as reliably on the build agent as it does locally. 5. Improve the test code if and where needed. Run the test locally AND through the pipeline after every change you make to get feedback on the impact of your code change. This feedback loop should still be VERY short, as we’re still working with a single test (or a very small group of tests, at the most). 6. Consider adding a linter for your test code. This is an optional step, but one I do recommend. At some point, you’ll probably want to enforce a common coding style anyway, and introducing a linter early on is way less painful. Consider being pretty strict. Warnings are nice and gentle, but easy to ignore. Errors, not so much. 7. Only after you’ve completed all the previous steps you can start adding more tests. All these new tests will now be linted, put under version control and be run locally and on a build agent, because you made that part of the process early on, thereby setting yourself up for success in the long term. 8. Make refactoring and optimizing your test code part of the process. Practices like (A)TDD have this step built in for a reason. 9. Once you’ve added a few more tests, start running them in parallel. Again, you want to start doing this early on, because it’s much harder to introduce parallelisation after you’ve already written hundreds of tests. 10 - ∞ Rinse and repeat. Forget about ‘building a test automation framework’. That ‘framework’ will emerge pretty much by itself as long as you stick to the process I outlined here and don’t skip the continuous refactoring.

Data engineers, have you ever written test cases that only cover specific, hardcoded inputs? You might feel confident that your code works, but what happens when it encounters an edge case you didn’t anticipate? Traditional testing can leave gaps, especially when dealing with dynamic data like API responses or user inputs. Imagine having tests that automatically cover a wide range of scenarios, including those tricky edge cases. With property-based testing, you can generate diverse test cases that push your code to its limits, ensuring it performs reliably under various conditions. This approach can dramatically increase the robustness of your code, giving you more confidence in its correctness. Enter the `hypothesis` library in Python. Instead of manually writing test cases for every possible input, `hypothesis` generates a wide range of inputs for you, systematically exploring your code’s behavior. 1. Traditional Test Case (left side): Here’s a typical `pytest` test for a `transform` function that adds a URL to a list of exchanges: This works for specific inputs, but what about other cases? What if the list is empty, or the exchange names are unusually long? A single test case won’t cover all possibilities. 2. Property-Based Testing with `hypothesis` (right side): With `hypothesis`, we can generate varied inputs to ensure the `transform` function handles them correctly. The Benefits: 1. Comprehensive Coverage: This approach ensures your code is tested against a wide range of inputs, catching edge cases you might miss with traditional tests. 2. Increased Confidence: You can trust that your code is robust and ready for production, no matter what data it encounters. 3. Efficiency: Property-based tests can replace dozens of manual test cases, saving time while increasing coverage. Property-based testing with `hypothesis` is a game-changer for data engineers. By automating the creation of diverse test cases, you ensure your code is reliable, robust, and production-ready. #dataengineering #python #propertybasedtesting #hypothesis #unittesting #techtips

I just published a video on the single most impactful change we’ve made to improve the quality of Playwright code generated by AI. Like a lot of teams, we leaned heavily into AI for writing test automation. And like a lot of teams, we ran into the same problem over time: drift. The code starts clean, then gradually becomes inconsistent, flaky, and harder to maintain. We tried the usual approaches. Better prompts, stricter rules, more guidance. They helped, but none of them actually solved the root issue. What did work was introducing what we call policy as code. Instead of relying on AI to “do the right thing,” we define a deterministic set of rules that represent our standards, architecture, and best practices. Then we run that rule set against every AI-generated change. The impact has been significant. Higher quality code, far less manual review, and most importantly, we catch bad patterns before they spread through the codebase. It essentially turns AI from something that needs constant supervision into something that can operate within clear, enforceable boundaries. If you’re using AI to generate Playwright tests, or really any code, I think this approach is worth looking into. I walk through the full setup and examples in the video. Let me know if you’ve run into similar issues or found other ways to control AI-generated code. https://lnkd.in/edDJgeqK

MCP Server Delivers AI-Generated Unit Tests and Advanced Fuzz Testing The server generates intelligent unit tests with proper edge cases, performs AI-powered fuzz testing to identify potential crashes, and conducts advanced coverage testing for maximum code path analysis. Each function receives 4-6 test cases while boundary testing uses 20 diverse inputs to probe system limits. The server combines BAML's structured generation with Gemini's language understanding, built on the FastMCP framework. It performs AST-based code analysis to detect branches, loops, and exception paths while integrating coverage.py for real-time reporting. The modular architecture allows teams to extend testing capabilities as needed. Software reliability becomes measurable and achievable at scale. Automated testing reduces manual QA overhead while catching edge cases that human testers might miss. Development cycles accelerate without sacrificing code quality, making robust software testing accessible to teams of any size. Vaibhav Gupta 👩💻https://lnkd.in/eHqRAJ38