Baidu released ERNIE-5.5-Flash-Base as an open base model. It is a 280B-parameter sparse Mixture-of-Experts model. It activates 14B parameters per token. It was pre-trained from scratch on 33.5T tokens. Training used a context window of up to 128K tokens. Fine-tuning and inference support a 1M-token context. The release includes no mid- or post-training. The architecture uses hybrid sparse attention and latent MoE. It also introduces manifold-constrained hyper-connections and a MultiMax output head. The tokenizer uses right-to-left grouping for numbers. The model is available on HuggingFace.
What changed
No prior open base model from Baidu combined hybrid sparse attention, latent MoE, and 1M-token context support.
What it unlocks
Developers can fine-tune or further train a high-capability base model without starting pre-training from scratch.
- 280B total parameters
- 14B activated per token
- 33.5T tokens pre-trained
- 128K training context
What you need to act on it
- access to HuggingFace or Baidu's model repository
- compute resources for 280B-parameter model inference or fine-tuning
Sources