Thanks for your questions! You can flexibly adjust the inference setup based on your computational resources without affecting results.

If you only want to test a subset, you can directly filter the query file, but for a small-scale experiment we don’t recommend taking the first 100, please sample randomly instead.

In the repo, we provide two evaluation approaches: using a trained critic model or external SOTA models as judges. To stay consistent with our latest leaderboard, we recommend using the same judge setup as the current leaderboard (currently Claude-4.5).

Originally posted by @A-Quarter-Mile in #19

Is it possible to modify the code to support using vLLM for acceleration during evaluation (by using the critic model trained in the paper)?