mlsanityData to 7 Oct 2026

Model Equality Testing: Which Model Is This API Serving?

arXiv 2410.20247 ↗First posted 26 Oct 2024cs.LGcited by 44

Not on the map. Why?

Abstract (arXiv)

Users often interact with large language models through black-box inference APIs, both for closed- and open-weight models (e.g., Llama models are popularly accessed via Amazon Bedrock and Azure AI Studio). In order to cut costs or add functionality, API providers may quantize, watermark, or finetune the underlying model, changing the output distribution -- possibly without notifying users. We formalize detecting such distortions as Model Equality Testing, a two-sample testing problem, where the user collects samples from the API and a reference distribution and conducts a statistical test to see if the two distributions are the same. We find that tests based on the Maximum Mean Discrepancy between distributions are powerful for this task: a test built on a simple string kernel achieves a median of 77.4% power against a range of distortions, using an average of just 10 samples per prompt. We then apply this test to commercial inference APIs from Summer 2024 for four Llama models, finding that 11 out of 31 endpoints serve different distributions than reference weights released by Meta.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Oct 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 27.

44 papers.

Sort:

By area:

  1. 2025Are You Getting What You Pay For? Auditing Model Substitution in LLM APIscited by 35
  2. 2025Cost-of-Pass: An Economic Framework for Evaluating Language Modelscited by 35
  3. 2025Idiosyncrasies in Large Language Modelscited by 32
  4. 2025Auditing Black-Box LLM APIs with a Rank-Based Uniformity Testcited by 16
  5. 2025SoK: Large Language Model Copyright Auditing via Fingerprintingcited by 14
  6. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  7. 2026Real Money, Fake Models: Deceptive Model Claims in Shadow APIscited by 8
  8. 2025Are Robust LLM Fingerprints Adversarially Robust?cited by 7
  9. 2025Natural Fingerprints of Large Language Modelscited by 6
  10. 2025Verifying LLM Inference to Detect Model Weight Exfiltrationcited by 6
  11. 2026LLM Spirals of Delusion: A Benchmarking Audit Study of AI Chatbot Interfacescited by 6
  12. 2026One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributionscited by 6
32 more
  1. 2025Pay for The Second-Best Service: A Game-Theoretic Approach Against Dishonest LLM Providerscited by 5
  2. 2026Token-Efficient Change Detection in LLM APIscited by 4
  3. 2025AWM: Accurate Weight-Matrix Fingerprint for Large Language Modelscited by 3
  4. 2026IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computationcited by 3
  5. 2026KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditingcited by 3
  6. 2026FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequencescited by 3
  7. 2026Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Servicescited by 3
  8. 2026What Iterated Self-Feeding Probes of Language Models Measure, and a test that separates the construction from the modelcited by 3
  9. 2025DiFR: Inference Verification Despite Nondeterminismcited by 2
  10. 2026AEX: Non-Intrusive Multi-Hop Attestation and Provenance for LLM APIscited by 2
  11. 2026Echoes within the Reasoning: Stealthy and Effective Watermarking via Chain of Thoughtcited by 2
  12. 2026Black-Box Forensics for Conversational LLM Agentscited by 2
  13. 2025Position: It's Time to Act on the Risk of Efficient Personalized Text Generationcited by 1
  14. 2025Minimax-Optimal Two-Sample Test with Sliced Wassersteincited by 1
  15. 2025One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMscited by 1
  16. 2026What LLMs Think When You Don't Tell Them What to Think About?cited by 1
  17. 2026Computational Arbitrage in AI Model Marketscited by 1
  18. 2026Beyond Via: Analysis and Estimation of the Impact of Large Language Models in Academic Paperscited by 1
  19. 2026Total Variation Distance Estimation in Autoregressive Modelscited by 1
  20. 2026Item Response Theory for AI Safetycited by 1
  21. 2025Kernel Quantile Embeddings and Associated Probability Metricscited by 0
  22. 2025RoBiologyDataChoiceQA: A Romanian Dataset for improving Biology understanding of Large Language Modelscited by 0
  23. 2026Rewriting the Response Path: Silent Tampering and Provider-Signed Defense in BYOK LLM Agentscited by 0
  24. 2026Referential Security as a New Paradigm for AI Evaluationscited by 0
  25. 2026Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuningcited by 0
  26. 2026Measuring Obedience to Authority Across Large Language Models with the Milgram Paradigmcited by 0
  27. 2026Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIscited by 0
  28. 2026AgentProv: Auditing Agentic LLM API Providers via Tool-use Policy Probescited by 0
  29. 2026A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Qualitycited by 0
  30. 2026Fingerprinting Multimodal Large Language Modelscited by 0
  31. 2026Who Is Behind the Harness? Fingerprinting LLMs through Agentic Behaviorcited by 0
  32. 2026Forging LLM Authorship Fingerprints with Targeted Rewritingcited by 0