Proposal to add tests + coverage report to pioasm #2789
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces run_tests.py, a Python script for automated validation of pioasm test cases. Updates all .pio test files with new output sections reflecting current pioasm command results, including error messages and generated code for various instruction, directive, and format scenarios.
Introduces a Bazel py_test target for running pioasm tests via unittest. Refactors run_tests.py to use unittest, auto-discovers test cases, and resolves PIOASM_BIN from environment or build output. Updates .pio test files to expect pioasm version 2.2.1-develop and adjusts output format error handling in test_output_formats.pio.
Introduces a new 'json_output' library to the Bazel build and integrates it into the pioasm binary. Updates multiple PIO test cases to reflect improved error handling and output, including successful JSON output generation and revised test programs for clock dividers, delay values, expressions, FIFO configuration, and output formats.
Moves and reorganizes test files into 'errors', 'valid', and 'edge' subdirectories for better coverage and clarity. Adds new error and validation test cases, updates Bazel test glob to include all nested test files, and removes obsolete or redundant tests. This improves test coverage for both valid and error conditions in the pioasm assembler.
Corrects test expectations for wrap configuration and Ada output in pioasm tests. Updates expected output to reflect successful compilation and generated code, replacing previous error messages.
Introduces coverage tests for pioasm core with new Bazel targets, shell scripts, and a C++ test harness. Updates test infrastructure to support coverage collection and output format validation, and fixes the Go output format test to expect success.
Replaces shell scripts with Python equivalents for test and coverage execution in tools/pioasm. Updates Bazel BUILD rules to use py_test and py_binary for improved cross-platform compatibility and maintainability. Adds documentation for running tests and generating coverage reports.
Introduces the outputs_match function to allow flexible comparison of test outputs, ignoring lines starting with '//?'. Updates run_test to use this function for more robust test validation.
Updated run_tests.py to preserve lines starting with '//?' when overwriting expected output in test files. Updated all test .pio files to use '//?' as a wildcard for the autogenerated version line, improving test stability against version changes.
Introduces new test files for pioasm covering various error conditions (e.g., invalid FIFO usage, instruction limits, symbol errors, sideset configuration, pin config, and more) and valid output scenarios for expressions and language-specific symbol generation (Go, JSON, Python). These tests improve coverage and validation of pioasm's error handling and output formatting.
Added new test cases to cover invalid character errors, invalid .lang_opt usage, unknown directive errors, and a valid JSON output scenario for pioasm. These tests improve coverage of error handling and output formatting.
Replaces error() with throw syntax_error when 'rel' is used with 'irq prev' or 'irq next' instructions in both parser.cpp and parser.yy. Adds tests to verify correct error reporting for these cases. This fixes a segfault on the two tests added in this commit as well.
Introduce new test cases to verify pioasm's error handling for various invalid syntax scenarios, including missing operands, unexpected tokens, unsupported modes, and incomplete instructions. These tests help ensure robust error reporting in the assembler.
Added new test cases for Ada, C SDK, JSON, and Python output formats, including error and valid scenarios. Updated expected output files for Ada, Go, and Python tests to reflect changes in autogenerated file headers.
Introduces a new test output file for validating the Python code generation of the pioasm tool with comprehensive PIO instructions. This file serves as a reference for correct output when running 'pioasm -o python' on a full-featured input.
Introduces new test files for pioasm covering valid and invalid configurations for clock dividers, pin counts, FIFO settings, mov status, and comprehensive config directives. These tests improve coverage of error handling and valid directive combinations for PIO assembly programs.
…/gonzalolarralde/pico-sdk into gonzalolarralde/add-pioasm-tests
Deleted obsolete or redundant test cases for clock divider and pin config errors. Also removed the 'gpio_v1_overflow' program from the GPIO range error tests to streamline the error test suite.
Added a test to verify that floating point values in .define statements produce syntax errors, and a test to confirm that various valid expressions are accepted and correctly evaluated in .define statements.
Extended test_output_json_full_valid.pio with .lang_opt directives and language-specific code blocks to test handling of custom output types. Updated expected warnings for unknown code block output types. Minor formatting change in test_define_float_errors.pio.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello 👋
I’m working on a small PIO compiler as a side project and needed a validation suite to ensure output alignment across implementations. This PR adds a snapshot-style test suite for pioasm that exercises most of the functionality by running real pioasm invocations and comparing captured output. The large diff is mostly test data rather than harness code.
Goals:
Coverage:
Notes:
I used LLMs to help generate test cases and harness code, but reviewed them for coherence and relevance.
Please let me know if this doesn’t align with the maintainers’ testing direction. I’m happy to adjust the approach, and completely understand if the decision is not to merge this PR.
Thanks for taking the time to review this!