Ai2 released Olmo-core 3, a redesigned open training stack for mixture-of-experts models. It scales MoE training into the trillion-parameter range while keeping active parameters per token roughly fixed. The stack switches from fully sharded data parallelism to distributed data parallelism. Expert parallelism, pipeline parallelism and a distributed optimizer split model and training state across GPUs. Row-wise expert parallelism, GPU-resident routing and grouped GEMM reduce routing and computation costs. MXFP8 support cuts computation and data movement while maintaining accuracy. Benchmarks show 2.7× throughput improvement on eight B300 GPUs for a 47B-parameter MoE. A 1.2T-parameter model achieved 858 TFLOP/s/GPU across 512 GPUs. DeepEP v2 enabled a 2.38T-parameter configuration in short-capacity tests. The report notes token gerrymandering, learning rate adjustments and overlapping communication pitfalls. Olmo-core 3 is open-source and will underpin the next Olmo generation.

• 8 to 128 experts, 4 active per token
• 3.2B active parameters per token
• 4.6B to 47B total parameter capacity
• <5% training throughput drop