Induction Labs, a two-person San Francisco team, published research introducing "imagination models" — a foundation model architecture that pretrains on internet video by predicting future frames as latent tokens rather than pixels, with no action labels. Its first test, Photon-1, is a sparse 106B-A5B MoE transformer pretrained from scratch for one epoch on 575 million frames (552 billion tokens, roughly 18 years of video at 1 fps) drawn from ~2 million computer screen recordings filtered from an internal index of 2 billion videos. Pretraining took about 30,000 H200 GPU-hours (4.4×10²² FLOPs) at 32K context with 40% MFU. A vision encoder using finite scalar quantization compresses each frame to 960 discrete tokens (~2.2KB).
Sources