Cohere released a serving engine for its North Mini Code model built around a single large GPU program. The design runs a whole decode step as one persistent kernel instead of dozens of small ones. Cohere says this cuts the waiting between kernel launches, which dominates small-batch decoding. It reports faster token generation than vLLM on a single H100 chip, with no measurable accuracy loss. The margin holds across batch sizes and out to 256K of context. The engine supports continuous batching, paged attention and ragged sequence lengths. It exposes an OpenAI-compatible endpoint with tool calling. The code is published on GitHub as a single CUDA file. Cohere credits Stanford's Hazy Research for the original task-list and counter-based synchronisation ideas. It also says it dropped shared-memory paging after finding the bookkeeping buggy and costly.

• 292 tok/s at batch size 1
• 1.58× faster than vLLM
• 185 tok/s for vLLM
• 62% of speed-of-light bandwidth