Researchers Jonah Leshin, Manish Shah, Ian Timmis and Daniel Kang posted a four-page arXiv paper (2603.19022, submitted 19 March 2026) describing Stability Monitor, a black-box system for detecting silent behavioral change in LLM API endpoints. The method periodically fingerprints an endpoint by sampling outputs from a fixed prompt set and comparing output distributions over time, using a summed energy distance statistic across prompts with permutation-test p-values aggregated sequentially to flag change events and define stability periods. The authors argue uptime, latency and throughput miss shifts caused by weight updates, tokenizer changes, quantization, inference engines, kernels, caching, routing or hardware. Controlled tests detected changes to model family, version, inference stack, quantization and behavioral parameters; real-world monitoring of the same model across multiple hosting providers showed substantial provider-to-provider and within-provider stability differences.

• arxiv.org2026-08-03