Researchers introduced ModAR, a world-action model that denoises multiple future modalities before predicting actions. It conditions each prediction on previously generated modalities. ModAR benefits from predicting point tracks, DINO features and depth maps. Predicting future RGB does not consistently improve performance. Sequential generation in ModAR outperforms existing world-action model formulations. It achieves a higher average success rate than Flex-π on the same data. ModAR uses roughly 20 times fewer training FLOPs and no pretraining. It outperforms baselines on three real-world bimanual tasks. Performance improves when trained with human videos.
What changed
World-action models predicted future observations and actions as RGB images only.
What it unlocks
Autoregressively denoising multiple future modalities before predicting actions in robotics tasks.
- 75% success rate
- 72% success rate for Flex-π
- 20× fewer training FLOPs
Sources