Meta deployed its Catching JiTTests system into production. It uses LLMs to generate executable tests that reveal bugs in code changes before they land. The system runs two workflows: dodgy diff and intent-aware. It filters false positives with an LLM-as-Judge ensemble and a rule-based assessor. Human engineers review only high-ranked catches. 589 Meta engineers have engaged with the system. It caught 74 serious privacy bugs in early 2026. Privacy SEVs at Meta fell by 20% in that period. The approach scales with a human-on-the-loop model. Meta published a paper on the deployment at FSE 2026.
What changed
Traditional testing and manual code reviews were the primary methods for catching bugs before production.
What it unlocks
Catching bugs in code changes before they land in production using AI-generated executable tests.
- 589 engineers engaged
- 60% true-positive rate
- 74 serious privacy bugs caught
- 20% reduction in privacy SEVs
What you need to act on it
- Meta's codebase and infrastructure
- LLM-based test generation and evaluation workflows
Sources