Researchers introduced PixelUMM, a model for image and video understanding and generation. It works directly in pixel space without a separate visual encoder. Images are represented as spatial patches and videos as spatiotemporal tubelets. A Mixture-of-Transformers architecture supports both understanding and generation tasks. The model jointly handles text prediction and pixel-space flow matching. Experiments show competitive performance across image and video tasks. The paper studies decoder design and spatial-temporal patch size choices. A project page accompanies the work.
What it unlocks
Building a single model that handles both image and video tasks without a separate visual encoder.
Sources