Double Ratchet co-evolves an inspectable evaluation metric with a lifecycle-managed skill library, so a self-improving LLM agent can be trained on tasks where no reliable metric exists to grade it. It is the reference implementation of Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents: the metric loop (evalratchet/) evolves a pool of drawback detectors and the expression that combines them, while the skill loop (vendor/skillevo/) evolves the agent's natural-language skills, both driven by a frozen LLM with no weight updates.

That skill loop is Ratchet, released on its own at Self-Evolving-Agents-Ratchet, with a copy pinned under vendor/ here so this repository runs standalone against one fixed version of it. Read that repository for the skill loop's own design and ablations, which assume a trustworthy metric is already given; read this one for the metric loop and the co-evolution driver that removes that assumption.

See Reproducibility for what is and is not included here, and Citation for the BibTeX entry.

evalratchet/        the metric loop: op pool, expression language, anchored
                    selection, lifecycle (synthesis / birth gate / retirement),
                    and the co-evolution driver
  ops/              per-task drawback-detector packs (mbpp, spider_sql, report_gen)
  synthesizer/      op + expression synthesis
  evolve.py         op lifecycle: birth gate, shadow tier, merit retirement
  storage/          metric/op stores (JSON/SQLite result files)
vendor/             the skill loop (Ratchet), pinned
  skillevo/         skill bank, router, capsules, curator, engine
  scripts/run_skill_loop.py   the skill-loop driver the runners call
scripts/            experiment entry points (one per loop x task) + suite adapters
config/config.py    single source of truth: models, roles, paths
datasets/           deterministic split builders (no data files included)
LICENSE             Apache-2.0 (covers this repo, including vendor/)

• Python >= 3.11; pip install pydantic pyyaml boto3
• Access to the Claude models named in config/config.py, served through Amazon Bedrock. The runtime uses the standard AWS SDK credential chain, so export AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY, set AWS_PROFILE, or run on an instance with an IAM role. Set MODEL_API_REGION to override the default region.
• Spider 2.0-Snow only: pip install snowflake-connector-python plus credentialed access to the Spider 2.0 hosted Snowflake warehouse. Place the official Spider2 repo (with its snowflake_credential.json) at ~/Spider2 or pass --spider2-repo / set SPIDER2_REPO. No credentials are stored in this repository; the connection factory reads that file at runtime and returns nothing if it is absent.
• Report generation only: this task needs report-generation data (tasks, evidence cards, golden sections) plus its RAQS evaluator, which are not included in this repository. Point the runner at your own copy via --report-skills or REPORT_SKILLS. The MBPP+ and Spider 2.0 tasks below run without it.

Run everything from this directory; the scripts put vendor/ on the path themselves.

No data files are included. Splits are built ONCE and stored; every experiment then reads the same stored files. The pipeline per task:

1. Screen (once): run the frozen solver 5x per task, grade with the official anchor, store per-task pass counts.

   python scripts/screen_mbpp.py baseline && python scripts/screen_mbpp.py subset
   python scripts/screen_spider2_snow.py          # needs warehouse access
   

   (Report generation needs no screening; its tasks derive from the golden demo reports in your report-generation data. Its dev soft labels come from scripts/label_report_gen_softlabels.py.)

2. Build the stored split (once): select the hard subset from the screen and freeze train / dev (eval_dev) / test (eval_locked) to datasets/<task>/splits/<task>.json (deterministic, seed 42). Each builder reads the screening output from step 1, so run these in order.

   For MBPP the two step-1 subcommands do this in sequence: baseline writes the per-task pass counts and subset selects the 60/40 hard split both loops share. build_splits.py then reads both files and adds the eval_dev anchor.

   python datasets/mbpp/build_splits.py
   python datasets/spider2/build_splits.py
   python datasets/report_gen/build_splits.py
   

3. Run experiments: every run script loads the stored split file. --seed varies only the loop's stochasticity (LLM sampling, synthesis order), never the task selection, so all runs and all seeds of a task share the exact same train/dev/test tasks. If the stored split is missing the loader falls back to a random split and prints a WARNING; that fallback does not reproduce the paper.

Every run writes results/<family>/<dataset>_<tag>/ with round-by-round loop.json, a final result.json, and sqlite stores; every number in the paper is computed from these. The paper uses three seeds per run (--seed).

Config defaults already pin every loop role to Claude Opus 4.7 as in the paper (the report final judge is Opus 4.8, set in final_judge_report_gen.py); the $M flag below makes that pinning explicit per run. Shared flags: M="--single-model global.anthropic.claude-opus-4-7", CURR="eval:15,skill:25,eval:8,skill:25,eval:5,skill:25,eval:2,skill:25".

Metric loop (Figure 2, Table 3):

python scripts/run_metric_evo.py          $M --rounds 100 --patience 100 --min-delta 0 --seed 42 --tag anchored
python scripts/run_metric_evo_spider2.py  $M --rounds 100 --patience 100 --min-delta 0 --seed 42 --tag anchored
python scripts/run_metric_evo_report.py   $M --rounds 100 --patience 100 --min-delta 0 --seed 42 --tag anchored

Ablation arms (Table 3): add --naive (anchor guards off) or --no-lifecycle (birth gate + retirement off; report script).

Reference skill loop (Figure 3 / Table 1, "skill" rows):

python scripts/run_skill_evo.py          $M --rounds 100 --patience 100 --min-delta 0 --skill-bank-max 50 --meta-every 10 --seed 42 --tag oracle
python scripts/run_skill_evo_spider2.py  $M --rounds 100 --skill-bank-max 50 --meta-every 10 --seed 42 --tag oracle
python scripts/run_skill_evo_report.py   $M --rounds 100 --patience 100 --min-delta 0 --skill-bank-max 50 --meta-every 10 --seed 42 --tag oracle

Double Ratchet co-evolution ("co" rows; add --naive for "co naive"):

python scripts/run_co_evo.py          $M --curriculum "$CURR" --skill-no-early-stop --seed 42 --tag co
python scripts/run_co_evo_spider2.py  $M --curriculum "$CURR" --skill-no-early-stop --seed 42 --tag co
python scripts/run_co_evo_report.py   $M --curriculum "$CURR" --skill-no-early-stop --skill-bank-max 50 --meta-every 10 --seed 42 --tag co

Final-judge 2x2 (Table 2), after the report runs finish:

python scripts/final_judge_report_gen.py --seeds 42,7,13                       # generic rubric
python scripts/final_judge_report_gen.py --seeds 42,7,13 --judge-tag taskaware # task-aware rubric
# --variant v3 selects the second-pass run directories

Each script's --help documents the remaining options; defaults reproduce the paper's configuration.

What is included. The full metric loop, the pinned skill loop, the co-evolution driver, every experiment entry point, the deterministic split builders, and the configuration that pins models and roles. The commands above are the ones the paper reports.

What is not included. No datasets, model outputs, run logs, result files, or credentials. Benchmark tasks are fetched from their official sources by the screening scripts, and splits are then derived locally, so nothing here pre-stages a dataset. The report-generation task additionally needs data and an evaluator that are not part of this repository, as noted under Requirements; the MBPP+ and Spider 2.0 tasks run without them.

Expected variance. Both loops call a hosted, non-deterministic LLM. --seed fixes task selection, split construction, and synthesis order, but not model sampling, so per-round numbers differ between runs of the same command. The paper's figures average three seeds (42, 7, 13); single-seed runs will not land on the reported values exactly. The Spider 2.0 task executes live queries against a hosted warehouse, which adds its own run-to-run variation and cost.

Environment. Python 3.11 or newer, with the packages listed under Requirements. Model access is through Amazon Bedrock in the region given by MODEL_API_REGION.

@misc{zhang2026whograder,
  title         = {Who Grades the Grader? Co-Evolving Evaluation Metrics and
                   Skills for Self-Improving LLM Agents},
  author        = {Xing Zhang and Guanghui Wang and Yanwei Cui and Ziyuan Li
                   and Wei Qiu and Bing Zhu and Peiyang He},
  year          = {2026},
  eprint        = {2607.12790},
  archivePrefix = {arXiv},
  url           = {https://arxiv.org/abs/2607.12790}
}

See CONTRIBUTING and our Code of Conduct.

See CONTRIBUTING for more information.

This project is licensed under the Apache-2.0 License.