A new inference engine performs parallel constrained decoding for structured JSON extraction. It evaluates all schema fields at once instead of token-by-token. On an Apple Silicon M4 Max it cuts latency by 5.6x to 7.0x. It guarantees 100% valid JSON output. It also provides calibrated confidence scores for each field. The engine uses KV-cache broadcasting to run all fields in a single forward pass. It supports up to 255 choices per enum field. A live demo and benchmark runner are available on Hugging Face Spaces. The code and SDK are open source under Apache 2.0. It currently targets Qwen2.5-1.5B-Instruct-4bit on MLX.
What changed
Structured generation ran token-by-token with linear latency scaling and risk of syntax errors.
What it unlocks
Extracting multi-field JSON from a prompt in one forward pass with guaranteed validity.
- 5.6x to 7.0x latency reduction
- 100% schema validity
- 255 max enum choices per field
- 1 forward pass for all fields
What you need to act on it
- Apple Silicon Mac (M1-M4)
- macOS 14.0+
- Python 3.10+
- mlx-lm supported model
Sources