# Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test

arXiv: https://arxiv.org/abs/2506.06975 · HTML: https://mlsanity.com/paper/2506.06975/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Builds on (most cited first, 23 of 23)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2306.05540/) (2023)
- [A Robust Semantics-based Watermark for Large Language Model against Paraphrasing](https://mlsanity.com/paper/2311.08721/) (2023)
- [Instructional Fingerprinting of Large Language Models](https://mlsanity.com/paper/2401.12255/) (2024)
- [LLMmap: Fingerprinting For Large Language Models](https://mlsanity.com/paper/2407.15847/) (2024)
- [Model Equality Testing: Which Model Is This API Serving?](https://mlsanity.com/paper/2410.20247/) (2024)
- [Are You Getting What You Pay For? Auditing Model Substitution in LLM APIs](https://mlsanity.com/paper/2504.04715/) (2025)
- [Idiosyncrasies in Large Language Models](https://mlsanity.com/paper/2502.12150/) (2025)
- [Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards](https://mlsanity.com/paper/2501.07493/) (2025)
- [Auditing Prompt Caching in Language Model APIs](https://mlsanity.com/paper/2502.07776/) (2025)
- [LLMs Have Rhythm: Fingerprinting Large Language Models Using Inter-Token Times and Network Traffic Analysis](https://mlsanity.com/paper/2502.20589/) (2025)
- [EICopilot: Search and Explore Enterprise Information over Large-scale Knowledge Graphs with LLM-driven Agents](https://mlsanity.com/paper/2501.13746/) (2025)

## Built on by (most cited first, 18 of 18)

- [SoK: Large Language Model Copyright Auditing via Fingerprinting](https://mlsanity.com/paper/2508.19843/) (2025)
- [Are Robust LLM Fingerprints Adversarially Robust?](https://mlsanity.com/paper/2509.26598/) (2025)
- [Verifying LLM Inference to Detect Model Weight Exfiltration](https://mlsanity.com/paper/2511.02620/) (2025)
- [One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions](https://mlsanity.com/paper/2607.10252/) (2026)
- [IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation](https://mlsanity.com/paper/2602.22700/) (2026)
- [KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditing](https://mlsanity.com/paper/2605.29524/) (2026)
- [DiFR: Inference Verification Despite Nondeterminism](https://mlsanity.com/paper/2511.20621/) (2025)
- [Total Variation Distance Estimation in Autoregressive Models](https://mlsanity.com/paper/2607.19510/) (2026)
- [Token Inflation: How Dishonest Providers Can Overcharge for Large Language Model Usage](https://mlsanity.com/paper/2605.30040/) (2026)
- [Bounded Behavioral Indistinguishability for Black-Box LLM Distillation](https://mlsanity.com/paper/2605.30448/) (2026)
- [RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studies](https://mlsanity.com/paper/2603.11001/) (2026)
- [Stemma: Induced Decision Regions Reveal LLM Provenance](https://mlsanity.com/paper/2607.25880/) (2026)
- [Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs](https://mlsanity.com/paper/2608.16391/) (2026)
- [Not to Break, but to Attest: Adversarial Probes for Privacy-Preserving LLM Verification](https://mlsanity.com/paper/2608.27954/) (2026)
- [Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behavior](https://mlsanity.com/paper/2609.28559/) (2026)
- [Strategic Self-Consistency](https://mlsanity.com/paper/2609.30352/) (2026)
- [Anosognosia in LLMs: Probing Self-Awareness of Quantized Computational Substrate](https://mlsanity.com/paper/2610.06174/) (2026)
- [Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting](https://mlsanity.com/paper/2610.17399/) (2026)

## Abstract (arXiv)

As API access becomes a primary interface to large language models (LLMs), users often interact with black-box systems that offer little transparency into the deployed model. To reduce costs or maliciously alter model behaviors, API providers may discreetly serve quantized or fine-tuned variants, which can degrade performance and compromise safety. Detecting such substitutions is difficult, as users lack access to model weights and, in most cases, even output logits. To tackle this problem, we propose a rank-based uniformity test that can verify the behavioral equality of a black-box LLM to a locally deployed authentic model. Our method is accurate, query-efficient, and avoids detectable query patterns, making it robust to adversarial providers that reroute or mix responses upon the detection of testing attempts. We evaluate the approach across diverse threat scenarios, including quantization, harmful fine-tuning, jailbreak prompts, and full model substitution, showing that it consistently achieves superior statistical power over prior methods under constrained query budgets.