Perplexity described the serving stack behind the search models that match queries to documents. Three components handle a request: a Rust gateway called Ivy, an inference server called Tulip, and an engine called ROSE. Ivy does text splitting and load-balances large batches across replicas. Tulip schedules requests first-come, first-served, since latency tracks token count more than request count. ROSE was originally built for large language models and now runs embedding models through the same code. Perplexity pre-records whole-model GPU instruction sequences to cut processor overhead on small batches. Capturing thousands of these takes minutes, so it happens gradually while serving. A LazyTensor abstraction lets the next batch start before the previous one finishes. Perplexity also upstreamed changes to FlashInfer to make the recorded sequences possible. It reports lower latency and higher throughput than vLLM in its own benchmarks.
What changed
Perplexity previously relied on off-the-shelf serving engines for embedding inference.
What it unlocks
Serving embedding and ranking models with lower latency by reusing an existing large-model serving stack.
- ~512 tokens saturates the GPU
- token counts padded to 64 or 256 buckets
- cosine divergence within 0.1%
- benchmarked against vLLM v0.22.0
What you need to act on it
- in-house GPU serving infrastructure
- engineering capacity to build custom inference components
Sources