mlsanityData to 14 Oct 2026

API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

arXiv 2609.08861 ↗First posted 8 Sep 2026cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

Benchmark scores are a central currency in model releases: they inform purchasing decisions, shape public trust, and influence policy. Yet, a key assumption underlying benchmark scores is that the model performance measured through APIs faithfully reflects the behavior of deployed systems. We challenge this assumption by auditing ChatGPT, Claude, and Gemini across seven systems and nine benchmarks spanning general capability, social bias, and sycophancy. We find systematic API--interface differences in both accuracy and consistency. On average, API evaluations score 3.4 percentage points higher in accuracy and 2.1 percentage points higher in test--retest agreement than corresponding interface evaluations. For ChatGPT, the performance difference between API and interface access rivals the API-only difference between GPT 5.3 and GPT 5.4. Put differently, switching access surfaces can degrade performance as much as downgrading a full model version. We further test whether exposed API controls can reproduce interface behavior by varying system prompts, sampling parameters, and reasoning settings. These controls shift behavior in some cases but do not reliably eliminate the gap. Our findings document a context-validity gap: measurements obtained through APIs do not necessarily generalize to corresponding deployed interfaces, complicating the use of API evaluations as proxies for deployed systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 59 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

34 of 59 references.

References · 34

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  9. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  10. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  11. 2024Measuring short-form factuality in large language modelscited by 373
  12. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
22 more
  1. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  2. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  3. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 195
  4. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  5. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 114
  6. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 102
  7. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 100
  8. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 65
  9. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  10. 2025Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotationcited by 50
  11. 2020Towards Ecologically Valid Research on Language User Interfacescited by 42
  12. 2023Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answeringcited by 41
  13. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  14. 2025ChatBench: From Static Benchmarks to Human-AI Evaluationcited by 34
  15. 2024RAD-Bench: Evaluating Large Language Models Capabilities in Retrieval Augmented Dialoguescited by 19
  16. 2024More than Marketing? On the Information Value of AI Benchmarks for Practitionerscited by 17
  17. 2025Fantastic Bugs and Where to Find Them in AI Benchmarkscited by 11
  18. 2025The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behaviorcited by 8
  19. 2025Auditing Google's AI Overviews and Featured Snippets: A Case Study on Baby Care and Pregnancycited by 7
  20. 2025A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harmscited by 5
  21. 2025DSA, AIA, and LLMs: Approaches to conceptualizing and auditing moderation in LLM-based chatbots across languages and interfaces in the electoral contextscited by 2
  22. 2026Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Promptscited by 2

Built on by

Not cited yet.