Researchers led by Guangyu Sun published FLAT, a method that puts images and text into one shared sequence. The same sequence can be used for search, for writing captions, and for generating images. A training trick orders the information from coarse to fine. Taking only the first part of the sequence trades detail for lower computing cost. The first token alone already captures the overall meaning of an image or caption. Longer prefixes recover layout, counting and finer attributes. Search accuracy barely moves between one token and the full 256. The authors also report untrained abilities, including Chinese and Spanish prompts, emoji prompts, and captions from video frames. Simple addition of tokens combines concepts, such as a moon added to a lighthouse scene. The paper is posted as a preprint.
What changed
Multimodal systems kept separate representations for understanding and for image generation.
What it unlocks
Using one shared sequence for search, captioning and image generation, with adjustable length.
- 83.1 GenEval
- 138.6 COCO CIDEr
- 81.8% ImageNet linear probe
- up to 256 tokens, 64 dims each
Sources