Verification in RSI Bench

Task Setup

RSI Bench aims to measure whether AI agents can advance AI R&D through open-ended, iterative research. A research-oriented task could be ill-designed in many ways: instructions that are under- or over-specified, verification that rewards only a subset of valid solutions, train/test splits that reward memorization over generalization, sensitivity to hyperparameter choices rather than genuine method improvements, or baselines that are not at the frontier. To this end, we designed a verification pipeline to ensure that every accepted task is well-designed to measure the intended capability.

Each task provides an agent with a fixed resource budget and a research environment containing a reproducible baseline and a validation feedback loop that allows agents to run multiple experiments, learn from their outcomes, and iterate on their ideas. The final submission is then measured by a separate hidden evaluator under held-out conditions (e.g., unseen data, models, or environments). If verification covers multiple objectives, it aggregates all scores into a final scalar reward.

The RSI Bench task setup. Inside a fixed budget, a researcher agent works in a workspace holding inputs, tools and a reproducible baseline, producing a submission that validation scores with val.sh; the agent iterates on that validation feedback. The final submission then goes once to a held-out evaluator running test.sh, whose metrics are aggregated into a single final reward: percentage improvement over the baseline. — RSI Bench task setup

Verification

RSI Bench uses Harbor task formats and builds on verification workflow developed for Terminal-Bench by extending it to this iterative-based research setting.

Contribute

RSI Bench is actively collecting tasks. You can explore this verification pipeline in our public GitHub repo. If you’re interested, please see our Call for Contributions.