Prime Intellect launched Prime Inference for resilient serving of frontier open-source models. It offers serverless and reserved capacity across multiple datacenters. The first public deployment is GLM-5.3 on OpenRouter. It supports automatic failover and OpenAI-compatible APIs. The stack uses NVIDIA Blackwell GPUs today. It combines NVIDIA Dynamo, vLLM, Mooncake, and FlashInfer. Prefill and decode run on separate GPU groups. NVFP4 KV compression increases cache capacity by 50%. A native NVFP4 kernel reduces decode latency. The service targets 100 tokens per second per user. It has delivered 100% uptime since launch. Production SLAs and 24/7 on-call support are included. Unified billing tracks team-level usage.
What changed
Frontier open-source models lacked resilient, production-grade serving infrastructure across multiple datacenters.
What it unlocks
Running frontier open-source models in production with serverless or reserved capacity, automatic failover, and OpenAI-compatible APIs.
- 100 tokens per second per user target
- 40% reduction in p90 inter-token latency
- 50% increase in cache capacity with NVFP4
- 66 sessions per prefill group at 101 tok/s/user
What you need to act on it
- an API key
- access to Prime Inference endpoints
Sources