Meta released aira-dojo, a framework for building and testing AI research agents. The code comes from Meta's paper on AI research agents for machine learning. It provides shared abstractions for tasks and for agents, called solvers. It implements the MLE-bench benchmark of machine-learning engineering tasks. Four agents from the paper are included, using greedy search, tree search and evolutionary search. An isolated code execution environment runs alongside job schedulers such as Slurm. Meta says the setup ran 1,000 agents at once for up to 120 hours. Use requires API keys for models such as o3 and GPT-4o. The licence is CC BY-NC 4.0, which bars commercial use.
What changed
The agents and search methods from the paper were described but not released as runnable code.
What it unlocks
Running and comparing machine-learning research agents on MLE-bench tasks at large scale on a cluster.
- 1,000 agents run in parallel
- up to 120 hours per run
- 166 stars, 36 forks
What you need to act on it
- conda and pip install
- LLM API keys and endpoints
- Apptainer container build
- Slurm cluster for parallel runs
- non-commercial CC BY-NC 4.0 licence
Sources