Researchers at Meta's FAIR lab built AIRA2, a system that runs machine-learning experiments on its own. It targets three limits found in earlier research agents. The first was slow throughput, because experiments ran one after another on a single GPU. AIRA2 spreads work across a pool of GPUs in isolated containers, so eight GPUs give roughly eight times the experiments. The second was agents tuning to their own scoring data and getting worse over long runs. A fixed, hidden evaluation split now supplies the score, and the labels are kept from the agent. The third was rigid, one-shot prompts for each step, which AIRA2 replaces with agents that choose their own actions and debug as they go. On MLE-bench-30, a set of Kaggle contests, it beat the strongest prior agent. It also passed the best published human results on some tasks in a second benchmark, AIRS-Bench. The team reports that the earlier decline over long runs came from noisy measurement, not memorised data.
What changed
Earlier research agents ran experiments one at a time on a single GPU, and their scores stopped improving after about 24 hours of search.
What it unlocks
Running automated machine-learning research runs that keep improving over three days instead of plateauing after one.
- 81.5% percentile rank at 24 hours
- 83.1% at 72 hours
- 72.7% for strongest baseline
- 6 of 20 AIRS-Bench tasks above human SOTA
What you need to act on it
- a multi-GPU cluster
- access to a strong underlying language model
Sources