# Model Equality Testing: Which Model Is This API Serving?

arXiv: https://arxiv.org/abs/2410.20247 · HTML: https://mlsanity.com/paper/2410.20247/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Oct 2024.

## Built on by (most cited first, 30 of 47)

- [Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs](https://mlsanity.com/paper/2504.04715/) (2025)
- [Cost-of-Pass: An Economic Framework for Evaluating Language Models](https://mlsanity.com/paper/2504.13359/) (2025)
- [Idiosyncrasies in Large Language Models](https://mlsanity.com/paper/2502.12150/) (2025)
- [Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test](https://mlsanity.com/paper/2506.06975/) (2025)
- [SoK: Large Language Model Copyright Auditing via Fingerprinting](https://mlsanity.com/paper/2508.19843/) (2025)
- [Predicting the Performance of Black-box LLMs through Follow-up Queries](https://mlsanity.com/paper/2501.01558/) (2025)
- [Are Robust LLM Fingerprints Adversarially Robust?](https://mlsanity.com/paper/2509.26598/) (2025)
- [Real Money, Fake Models: Deceptive Model Claims in Shadow APIs](https://mlsanity.com/paper/2603.01919/) (2026)
- [Natural Fingerprints of Large Language Models](https://mlsanity.com/paper/2504.14871/) (2025)
- [Pay for The Second-Best Service: A Game-Theoretic Approach Against Dishonest LLM Providers](https://mlsanity.com/paper/2511.00847/) (2025)
- [Verifying LLM Inference to Detect Model Weight Exfiltration](https://mlsanity.com/paper/2511.02620/) (2025)
- [LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfaces](https://mlsanity.com/paper/2604.06188/) (2026)
- [One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions](https://mlsanity.com/paper/2607.10252/) (2026)
- [Token-Efficient Change Detection in LLM APIs](https://mlsanity.com/paper/2602.11083/) (2026)
- [AWM: Accurate Weight-Matrix Fingerprint for Large Language Models](https://mlsanity.com/paper/2510.06738/) (2025)
- [IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation](https://mlsanity.com/paper/2602.22700/) (2026)
- [AEX: Non-Intrusive Multi-Hop Attestation and Provenance for LLM APIs](https://mlsanity.com/paper/2603.14283/) (2026)
- [KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing](https://mlsanity.com/paper/2605.29524/) (2026)
- [FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequences](https://mlsanity.com/paper/2606.03330/) (2026)
- [Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Services](https://mlsanity.com/paper/2606.16100/) (2026)
- [What Iterated Self-Feeding Probes of Language Models Measure, and a test that separates the construction from the model](https://mlsanity.com/paper/2608.10986/) (2026)
- [DiFR: Inference Verification Despite Nondeterminism](https://mlsanity.com/paper/2511.20621/) (2025)
- [Echoes within the Reasoning: Stealthy and Effective Watermarking via Chain of Thought](https://mlsanity.com/paper/2605.28890/) (2026)
- [Black-Box Forensics for Conversational LLM Agents](https://mlsanity.com/paper/2606.22698/) (2026)
- [Total Variation Distance Estimation in Autoregressive Models](https://mlsanity.com/paper/2607.19510/) (2026)
- [Position: It's Time to Act on the Risk of Efficient Personalized Text Generation](https://mlsanity.com/paper/2502.06560/) (2025)
- [Minimax-Optimal Two-Sample Test with Sliced Wasserstein](https://mlsanity.com/paper/2510.27498/) (2025)
- [One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs](https://mlsanity.com/paper/2512.14751/) (2025)
- [What LLMs Think When You Don't Tell Them What to Think About?](https://mlsanity.com/paper/2602.01689/) (2026)
- [Computational Arbitrage in AI Model Markets](https://mlsanity.com/paper/2603.22404/) (2026)

## Abstract (arXiv)

Users often interact with large language models through black-box inference APIs, both for closed- and open-weight models (e.g., Llama models are popularly accessed via Amazon Bedrock and Azure AI Studio). In order to cut costs or add functionality, API providers may quantize, watermark, or finetune the underlying model, changing the output distribution -- possibly without notifying users. We formalize detecting such distortions as Model Equality Testing, a two-sample testing problem, where the user collects samples from the API and a reference distribution and conducts a statistical test to see if the two distributions are the same. We find that tests based on the Maximum Mean Discrepancy between distributions are powerful for this task: a test built on a simple string kernel achieves a median of 77.4% power against a range of distortions, using an average of just 10 samples per prompt. We then apply this test to commercial inference APIs from Summer 2024 for four Llama models, finding that 11 out of 31 endpoints serve different distributions than reference weights released by Meta.