Allen Institute for AI released a redesigned Olmo-core training stack for Mixture-of-Experts models. It replaces Fully Sharded Data Parallel with Distributed Data Parallel. The stack adds Expert Parallelism, Pipeline Parallelism, a distributed optimizer, and activation recompute. NVSHMEM-based rowwise Expert Parallelism avoids permutation overhead in dispatch and combine. GPU-resident routing metadata removes host synchronization points. MXFP8 precision reduces compute, activation-memory, and transport costs. Topology-agnostic checkpoints allow parallel layout changes between runs. The stack supports 12.9B to 1.2T parameters on up to 512 GPUs. It achieves 858 TFLOP/s/GPU at 1.2T parameters with MXFP8. An experimental run reached 2.38T parameters. The report identifies Token Gerrymandering as a router failure pattern. Overlapping communication with computation can slow kernels more than it hides latency.

• 12.9B to 1.2T total parameters
• 512 GPUs max
• 858 TFLOP/s/GPU at 1.2T with MXFP8
• 2.38T parameters experimental