Researchers proved the Platonic Representation Hypothesis for JEPA models. They showed these models learn the same representation across modalities when trained on good views. Next-token prediction was shown to be a JEPA with bad views. Good views required 42 times fewer samples than next-token prediction for equal alignment. A 109M-parameter text JEPA outperformed language models up to 72B parameters in alignment. It also beat CLIP’s text encoder in alignment with image encoders. One rotation turned separate encoders into an image captioner with a .838 CIDEr score on COCO. The text JEPA achieved 64% top-5 accuracy on ImageNet-100 without paired data. This outperformed all zero-shot language model results up to 8B parameters. The work uncovered a scaling law predicting alignment across runs.
What changed
The Platonic Representation Hypothesis was unproven and lacked a practical method for implementation.
What it unlocks
Building aligned representations across modalities with far less data by using good views instead of next-token prediction.
- 42x fewer training samples for good views vs next-token prediction
- 109M-parameter text JEPA
- 72B parameters for compared language models
- 64% top-5 accuracy on ImageNet-100
What you need to act on it
- JEPA architecture
- good views (paired observations of the same scene)
- bounded misalignment
Sources