mlsanityData to 14 Oct 2026

Ventor-QTest: Threat-Model-Driven Verification of Vendor-Hosted LLM APIs

arXiv 2608.16391 ↗First posted 17 Aug 2026cs.CR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

As large language models become increasingly widespread, third-party providers that deploy open-weight models have become an important part of the ecosystem. Auditing the quality of their inference APIs is therefore an open problem. We formalize hosted model routing as a stochastic process and propose mboxVentor-QTest, a composite black-box audit that requires no probability information from the target API. Its repeated-request component sends each frozen constrained context to the target multiple times, reconstructs a categorical output distribution from the returned text counts, and reports average fidelity loss (AFL) as a null-bias-corrected, within-window mean coarsened-KL statistic. Its long-sequence component uses independent runs to report extreme fidelity loss (EFL) through the empirical upper tail of a run-level reference-centered-surprisal statistic. Across three logprob-capable route conditions, AFL shows strong linear descriptive agreement with a logprob-derived coarsened-KL comparator. Across seven route snapshots, 20-run sequence probes reveal route-specific EFL variation. AFL and EFL have little detectable route-level association with GPQA-Diamond accuracy. In contrast, pronounced EFL coincides with a decline in Terminal-Bench pass rate as task exposure increases. This pattern may arise because correctness in long-horizon tasks is more sensitive to extreme fidelity loss. These results motivate reporting AFL and EFL jointly, particularly when auditing long-horizon agentic tasks. The open-source implementation is available at https://github.com/Tencent/AI-Infra-Guard/tree/main/services/apichecker/ventorqtest.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 63 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

24 of 63 references.

References · 24

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  5. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  6. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  7. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  8. 2024LLMmap: Fingerprinting For Large Language Modelscited by 58
  9. 2025Optimizing Large Language Model Training Using FP4 Quantizationcited by 56
  10. 2024Model Equality Testing: Which Model Is This API Serving?cited by 47
  11. 2025Are You Getting What You Pay For? Auditing Model Substitution in LLM APIscited by 38
  12. 2025Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latencycited by 34
12 more
  1. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Modelscited by 22
  2. 2024Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluationcited by 19
  3. 2025Auditing Black-Box LLM APIs with a Rank-Based Uniformity Testcited by 18
  4. 2025Log Probability Tracking of LLM APIscited by 10
  5. 2026One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributionscited by 6
  6. 2026Token-Efficient Change Detection in LLM APIscited by 5
  7. 2025Economies of Open Intelligence: Tracing Power & Participation in the Model Ecosystemcited by 4
  8. 2026KBF: Knowledge Boundary as Fingerprint for Language Model and Black-Box API Auditingcited by 3
  9. 2026FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequencescited by 3
  10. 2026Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Servicescited by 3
  11. 2026Which Model Is Actually Serving You? IRIS: Budgeted Black-Box Auditing of Model Substitution and Routing Dilution in LLM Gatewayscited by 3
  12. 2025You've Changed: Detecting Modification of Black-Box Large Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety
  2. 2026AEX: Non-Intrusive Multi-Hop Attestation and Provenance for LLM APIs
  3. 2025Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy
  4. 2024vTune: Verifiable Fine-Tuning for LLMs Through Backdooring
  5. 2026Proxy Confidence: Auditing Black-Box LLM Agents with a Surrogate's Log-Probabilities
  6. 2026Membership Inference on LLMs in the Wild
  7. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defense
  8. 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines