-
Notifications
You must be signed in to change notification settings - Fork 677
Update SAGE to ATLAS & Update corresponding files #2297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR refactors the SAGE (Scientific Advanced General Evaluation) dataset implementation to ATLAS (A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning), involving a comprehensive rename of classes, functions, variables, and configuration files throughout the codebase.
Key changes:
- Renamed dataset classes, evaluator classes, and functions from SAGE to ATLAS naming convention
- Updated dataset loading to use 'opencompass/ATLAS' and changed the answer field from 'standard_answer' to 'refined_standard_answer'
- Added explicit string conversion for question and answer fields in the dataset loader
- Reorganized configuration files by removing the old sage configs and adding new atlas configs
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| opencompass/datasets/atlas/prompt.py | Newly added prompt templates, but uses incorrect SAGE_* variable names instead of ATLAS_* |
| opencompass/datasets/atlas/evaluation.py | Renamed functions and class from sage_*/SAGE* to atlas_*/ATLAS* |
| opencompass/datasets/atlas/dataset_loader.py | Renamed class to ATLASDataset, updated dataset path and field names with string conversions |
| opencompass/configs/datasets/sage/sage_gen.py | Removed old SAGE configuration entry point |
| opencompass/configs/datasets/sage/README.md | Removed old SAGE documentation |
| opencompass/configs/datasets/atlas/atlas_val_gen_906a48.py | Renamed all references from SAGE to ATLAS, includes duplicate prompt definitions |
| opencompass/configs/datasets/atlas/atlas_gen.py | New ATLAS configuration entry point |
| opencompass/configs/datasets/atlas/README.md | New ATLAS documentation |
Comments suppressed due to low confidence (2)
opencompass/configs/datasets/atlas/atlas_val_gen_906a48.py:11
- The prompt templates
ATLAS_INFER_TEMPLATEandATLAS_EVAL_TEMPLATEare duplicated in this configuration file (lines 10-36 and 38-164). They should be imported fromopencompass.datasets.atlas.promptinstead to avoid code duplication and maintain a single source of truth. Note that the prompt.py file has incorrect naming (SAGE_*) that needs to be fixed first.
opencompass/configs/datasets/atlas/atlas_val_gen_906a48.py:11 - Extra trailing whitespace after 'Problem:' on this line. Should be 'Problem:' without the trailing space.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| with read_base(): | ||
| from .atlas_val_gen_906a48 import atlas_datasets # noqa: F401, F403 No newline at end of file |
Copilot
AI
Oct 31, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Import of 'atlas_datasets' is not used.
| with read_base(): | |
| from .atlas_val_gen_906a48 import atlas_datasets # noqa: F401, F403 | |
| with read_base(): |
No description provided.