# Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs

arXiv: https://arxiv.org/abs/2504.04715 · HTML: https://mlsanity.com/paper/2504.04715/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Provable Robust Watermarking for AI-Generated Text](https://mlsanity.com/paper/2306.17439/) (2023)
- [Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature](https://mlsanity.com/paper/2310.05130/) (2023)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [RADAR: Robust AI-Text Detection via Adversarial Learning](https://mlsanity.com/paper/2307.03838/) (2023)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text](https://mlsanity.com/paper/2305.17359/) (2023)
- [A Survey on Detection of LLMs-Generated Content](https://mlsanity.com/paper/2310.15654/) (2023)
- [Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression](https://mlsanity.com/paper/2403.15447/) (2024)
- [zkLLM: Zero Knowledge Proofs for Large Language Models](https://mlsanity.com/paper/2404.16109/) (2024)
- [SoK: Watermarking for AI-Generated Content](https://mlsanity.com/paper/2411.18479/) (2024)
- [Model Equality Testing: Which Model Is This API Serving?](https://mlsanity.com/paper/2410.20247/) (2024)
- [Logits of API-Protected LLMs Leak Proprietary Information](https://mlsanity.com/paper/2403.09539/) (2024)
- [Idiosyncrasies in Large Language Models](https://mlsanity.com/paper/2502.12150/) (2025)
- [Towards Possibilities & Impossibilities of AI-generated Text Detection: A Survey](https://mlsanity.com/paper/2310.15264/) (2023)
- [Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards](https://mlsanity.com/paper/2501.07493/) (2025)
- [TOPLOC: A Locality Sensitive Hashing Scheme for Trustless Verifiable Inference](https://mlsanity.com/paper/2501.16007/) (2025)
- [Predicting the Performance of Black-box LLMs through Follow-up Queries](https://mlsanity.com/paper/2501.01558/) (2025)

## Built on by (most cited first, 30 of 38)

- [Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test](https://mlsanity.com/paper/2506.06975/) (2025)
- [SoK: Large Language Model Copyright Auditing via Fingerprinting](https://mlsanity.com/paper/2508.19843/) (2025)
- [CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs](https://mlsanity.com/paper/2505.13778/) (2025)
- [Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives](https://mlsanity.com/paper/2505.21627/) (2025)
- [Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Services](https://mlsanity.com/paper/2505.18471/) (2025)
- [Log Probability Tracking of LLM APIs](https://mlsanity.com/paper/2512.03816/) (2025)
- [Real Money, Fake Models: Deceptive Model Claims in Shadow APIs](https://mlsanity.com/paper/2603.01919/) (2026)
- [Predictive Auditing of Hidden Tokens in LLM APIs via Reasoning Length Estimation](https://mlsanity.com/paper/2508.00912/) (2025)
- [Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators](https://mlsanity.com/paper/2505.18601/) (2025)
- [Pay for The Second-Best Service: A Game-Theoretic Approach Against Dishonest LLM Providers](https://mlsanity.com/paper/2511.00847/) (2025)
- [One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions](https://mlsanity.com/paper/2607.10252/) (2026)
- [Token-Efficient Change Detection in LLM APIs](https://mlsanity.com/paper/2602.11083/) (2026)
- [Auditing Pay-Per-Token in Large Language Models](https://mlsanity.com/paper/2510.05181/) (2025)
- [Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism](https://mlsanity.com/paper/2601.06118/) (2026)
- [Test-Time Compute Games](https://mlsanity.com/paper/2601.21839/) (2026)
- [Proof-of-Guardrail in AI Agents and What (Not) to Trust from It](https://mlsanity.com/paper/2603.05786/) (2026)
- [IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation](https://mlsanity.com/paper/2602.22700/) (2026)
- [Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Services](https://mlsanity.com/paper/2606.16100/) (2026)
- [Total Variation Distance Estimation in Autoregressive Models](https://mlsanity.com/paper/2607.19510/) (2026)
- [TRUST: A Decentralized Framework for Auditing Large Language Model Reasoning](https://mlsanity.com/paper/2510.20188/) (2025)
- [Computational Arbitrage in AI Model Markets](https://mlsanity.com/paper/2603.22404/) (2026)
- [VOW: Verifiable and Oblivious Watermark Detection for Large Language Models](https://mlsanity.com/paper/2604.27666/) (2026)
- [Inferring the Size of Large Language Models From Popular Text Memorization](https://mlsanity.com/paper/2605.29223/) (2026)
- [Token Inflation: How Dishonest Providers Can Overcharge for Large Language Model Usage](https://mlsanity.com/paper/2605.30040/) (2026)
- [Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service](https://mlsanity.com/paper/2506.06446/) (2025)
- [Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibes](https://mlsanity.com/paper/2508.05469/) (2025)
- [Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served](https://mlsanity.com/paper/2603.02348/) (2026)
- [Adaptive Contracts for Cost-Effective AI Delegation](https://mlsanity.com/paper/2603.17212/) (2026)
- [Towards Verifiable AI with Lightweight Cryptographic Proofs of Inference](https://mlsanity.com/paper/2603.19025/) (2026)
- [Referential Security as a New Paradigm for AI Evaluations](https://mlsanity.com/paper/2605.25673/) (2026)

## Abstract (arXiv)

Commercial Large Language Model (LLM) APIs create a fundamental trust problem: users pay for specific models but have no guarantee that providers deliver them faithfully. Providers may covertly substitute cheaper alternatives (e.g., quantized versions, smaller models) to reduce costs while maintaining advertised pricing. We formalize this model substitution problem and systematically evaluate detection methods under realistic adversarial conditions. Our empirical analysis reveals that software-only methods are fundamentally unreliable: statistical tests on text outputs are query-intensive and fail against subtle substitutions, while methods using log probabilities are defeated by inherent inference nondeterminism in production environments. We argue that this verification gap can be more effectively closed with hardware-level security. We propose and evaluate the use of Trusted Execution Environments (TEEs) as one practical and robust solution. Our findings demonstrate that TEEs can provide provable cryptographic guarantees of model integrity with only a modest performance overhead, offering a clear and actionable path to ensure users get what they pay for. Code is available at https://github.com/sunblaze-ucb/llm-api-audit