Perplexity released Q2D-Web, a private benchmark with a public leaderboard for search retrieval. It measures how well embedding models find candidate documents from a large web corpus. The corpus holds about 190 million pages and the query set 69,721 agent-rewritten queries in ten languages. Queries come from nine months of production search traffic, filtered for personal information. Relevance is labelled three ways: agent citations, production web rankings, and added language-model judgments. No single model leads on all three label sets. Perplexity's own pplx-embed-v1-4b scores highest on two of them, and Nemotron-3-Embed-8B on citations. Perplexity notes its models may gain an advantage because the data comes from its own traffic. A smaller sampled corpus cuts evaluation cost to about a third while preserving model ranking. Anyone can submit a public Hugging Face model for evaluation.
What changed
The closest public comparison, MS MARCO Web Search, had 100.9 million documents, 9,374 test queries and one click-derived label per query.
What it unlocks
Comparing publicly released embedding models for first-stage web retrieval on a large corpus with three separate relevance-label sets.
- 190M documents, 69,721 queries
- 99.6 positive judgments per query
- pplx-embed-v1-4b: 69.11 Combined Recall@1000
- RRF subcorpus uses 31.7% of corpus
What you need to act on it
- a publicly available, non-gated Hugging Face model
- loads via standard transformers or sentence-transformers
- submission through Perplexity's evaluation request form
Sources