# Which Model Is Actually Serving You? IRIS: Budgeted Black-Box Auditing of Model Substitution and Routing Dilution in LLM Gateways

arXiv: https://arxiv.org/abs/2607.20860 · HTML: https://mlsanity.com/paper/2607.20860/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2026.

## Built on by (most cited first, 3 of 3)

- [What Iterated Self-Feeding Probes of Language Models Measure, and a test that separates the construction from the model](https://mlsanity.com/paper/2608.10986/) (2026)
- [Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs](https://mlsanity.com/paper/2608.16391/) (2026)
- [SLICE: Sequential Likelihood-Free Inference via Conformal E-Values](https://mlsanity.com/paper/2610.13855/) (2026)

## Abstract (arXiv)

Commercial LLM gateways mediate access to hosted models, but the served backend may not match the advertised one: it may substitute a cheaper model on every request or route only a fraction ε of requests to it. Prior black-box auditors often need a privileged signal (log-probabilities, token ranks, or reference samples) or a target-specific probe, fix the query budget in advance, and return a yes/no verdict. We present IRIS, an audit that needs only the returned text: it asks endpoints to generate random numbers or strings, fingerprints the backend, and is the first to combine, in one text-only audit, detection of whole-stream substitution and fractional dilution, attribution of the served backend, routing-fraction (ε) estimation, and a query budget it sizes itself. A cheap pilot fits the exponential query-error decay and freezes that budget before any suspect query is issued. On an intra-family Qwen3 ladder IRIS verifies the backend at 0.99 AUROC and sharpens attribution as queries accumulate; across a commercial OpenRouter library it catches ε=0.3 dilution on margin-qualified pairs at 0.85 mean power (0.017 false-positive rate) and recovers ε to within 0.04 for enrolled diluents; and a live cross-provider audit flags 14 of 15 same-model provider pairs by genuine quantization and kernel deviations, corroborated on third-party MET traces. Against comparable black-box auditors, IRIS matches or beats detection on shared tasks, and adaptive allocation lifts the matched-budget target-hit rate from 73% to 87%. Further experiments cover adversarial gateways, knob identifiability, unseen diluents, and false-positive control.