JSON report generation from scan results #1929
Conversation
|
@bmalezieux you might need something different, but for reference the AVID integration could be helpful (it also generates JSON reports from the scan, following the AVID schema). |
rabah-khalek
left a comment
rabah-khalek
left a comment
There was a problem hiding this comment.
since <detector>.run() produces a list of issues, which then you're looping over to jsonify, I would suggest to implicitly infer the detector names inside the run, and add this info to the issue's meta, which then you can group based on in to_json of ScanReport. detectors_names shouldn't be an input to the report, as we already have this info somewhere.
giskard/core/suite.py
Outdated
| def to_json(self, filename=None): | ||
| results = {} | ||
| for suite_result in self.results: | ||
| results[suite_result.test_name] = "Passed" if suite_result.result.passed else "Failed" |
There was a problem hiding this comment.
can we add all the meta data that is available / could be useful?
- metric value
- model name
- dataset(s) name(s)
giskard/core/suite.py
Outdated
| with open(filename, "w") as json_file: | ||
| json.dump(results, json_file, indent=4) | ||
| else: | ||
| return results |
There was a problem hiding this comment.
I would perhaps return the json representation instead of the dict here
return json.dumps(results, indent=4)There was a problem hiding this comment.
Or let's simply remove this else, and always return the results
giskard/core/suite.py
Outdated
|
|
||
| if isinstance(result, bool): | ||
| result = TestResult(passed=result) | ||
| print("TEMP: ", test_partial.test_id, result.passed) |
|
|
||
| class ScanReport: | ||
| def __init__(self, issues, model=None, dataset=None, as_html: bool = True): | ||
| def __init__(self, issues, model=None, dataset=None, detectors_names=None, as_html: bool = True): |
There was a problem hiding this comment.
i thought the idea was to remove this argument. Do we really need to know which detector didn't produce issues?
| assert dest.read_text() == markdown | ||
|
|
||
|
|
||
| def test_scan_report_to_json(): |
There was a problem hiding this comment.
let's add a test where you pass the detector-names list to the scanreport, and check if the detector is in the json file
|
Please retry analysis of this Pull-Request directly on SonarCloud |
Description
Related Issue
Type of Change
Checklist
CODE_OF_CONDUCT.mddocument.CONTRIBUTING.mdguide.pdm.lockrunningpdm update-lock(only applicable whenpyproject.tomlhas beenmodified)