mlsanityData to 14 Oct 2026

Response drift across frontier large language models

arXiv 2607.20454 ↗First posted 14 May 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

All frontier large language models (LLMs) exhibit response drift -- producing outputs that deviate from expert-validated references -- yet the magnitude and structure of this drift remain uncharacterised by systematic human evaluation. Here we report a fully crossed evaluation in which 47 geographically diverse participants each assessed all 62 multidomain questions across ten frontier LLMs under blinded conditions, yielding 29,140 independent assessments. Every model drifts, but drift magnitude varies substantially: eight models converge on a statistically indistinguishable ceiling (78-81% deviation), while two achieve lower deviation (47-49%). Drift profiles differ across six domains and 62 questions, with pairwise correlations among ceiling models exceeding r = 0.85. Automated similarity metrics explain less than 2% of variance in human judgements. These findings reveal that response drift is universal across frontier LLMs, domain- and question-dependent in structure, and accessible only through human-centred evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 48 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

24 of 48 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
12 more
  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  4. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  5. 2025Humanity's Last Examcited by 491
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  7. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  8. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  9. 2023Do Language Models Know When They're Hallucinating References?cited by 116
  10. 2024What Large Language Models Know and What People Think They Knowcited by 110
  11. 2024MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmarkcited by 41
  12. 2024Densing Law of LLMscited by 40

Built on by

Not cited yet.