mlsanityData to 14 Oct 2026

The Inadequacy of Offline LLM Evaluations: A Need to Account for Personalization in Model Behavior

arXiv 2509.19364 ↗First posted 18 Sep 2025cs.CL cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Standard offline evaluations for language models -- a series of independent, state-less inferences made by models -- fail to capture how language models actually behave in practice, where personalization fundamentally alters model behavior. For instance, identical benchmark questions to the same language model can produce markedly different responses when prompted to a state-less system, in one user's chat session, or in a different user's chat session. In this work, we provide empirical evidence showcasing this phenomenon by comparing offline evaluations to field evaluations conducted by having 800 real users of ChatGPT and Gemini pose benchmark and other provided questions to their chat interfaces.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 29 of its 37 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

29 of 37 references.

References · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  5. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  6. 2020Aligning AI With Shared Human Valuescited by 723
  7. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  8. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  9. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  10. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  11. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  12. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
17 more
  1. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  2. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 208
  3. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  4. 2025Position: Evaluating Generative AI Systems Is a Social Science Measurement Challengecited by 63
  5. 2025Toward an Evaluation Science for Generative AI Systemscited by 45
  6. 2020The Problem with Metrics is a Fundamental Problem for AIcited by 37
  7. 2023BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialoguescited by 34
  8. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 34
  9. 2025ChatBench: From Static Benchmarks to Human-AI Evaluationcited by 34
  10. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 27
  11. 2024The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cited by 22
  12. 2025Presumed Cultural Identity: How Names Shape LLM Responsescited by 21
  13. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20
  14. 2024First-Person Fairness in Chatbotscited by 20
  15. 2024A User-Centric Multi-Intent Benchmark for Evaluating Large Language Modelscited by 15
  16. 2025Evaluating the Sensitivity of LLMs to Prior Contextcited by 9
  17. 2025LLM Safety for Childrencited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026One Persona, Many Cues, Different Results: How Sociodemographic Cues Impact LLM Personalizationcited by 9
  2. 2026Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Biascited by 4
  3. 2026Do LLMs Favor LLMs? Quantifying Interaction Effects in Peer Reviewcited by 3
  4. 2026What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)cited by 1
  5. 2026Protecting patient privacy in clinical foundation models: Technical and legal perspectivescited by 1
  6. 2026From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-Worldcited by 0
  7. 2026Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Researchcited by 0
  8. 2026API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfacescited by 0