Anthropic released Petri, a free and openly available tool that automatically tests how an AI model behaves across many simulated conversations and then scores the results. It is aimed at safety researchers and model developers, who supply plain-language descriptions of the scenarios they want probed. Anthropic ran a pilot across 14 frontier models and cautions that the scoring measures are provisional and cover a small set of scenarios.
What changed
Building alignment evaluations meant manually constructing scenarios, running models and reading transcripts by hand.
What it unlocks
Testing a hypothesis about how a model behaves by writing a plain-language seed instruction and letting an automated agent run and score the conversations.
- 14 frontier models tested
- 111 seed instructions
What you need to act on it
- access to a model API
- engineering setup from the GitHub repository
Sources