mlsanityData to 7 Oct 2026

No Universal Signal Predicts Sample-Level LLM Regression under Version Updates

arXiv 2608.13607 ↗First posted 11 Aug 2026cs.AI cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Frontier LLMs are updated frequently and typically outperform their predecessors in aggregate. But aggregate gains say little about individual samples: an update can still cause sample-level regression, where a response correct under the old model becomes incorrect under the new one. This paper studies how to predict such regressions from signals available at inference time. We compare single-model signals (confidence, logit margin, attention entropy) against cross-version signals (output KL divergence, likelihood drift, token-level KL, representation drift) under a unified added-value test that isolates each signal's gain over a confidence baseline. Across six benchmarks in three task families (multiple-choice question answering, or MCQ; math reasoning; code generation) and six model update pairs, we find that (1) signal effectiveness is task-dependent: confidence is strongest on MCQ and simpler math, while likelihood/KL signals give the most frequent gains on harder math and code; (2) no signal is universally best across model updates either; and (3) some cross-version signals stay informative even when confidence fails, including without labels, which supports a proof-of-concept selective fallback that routes high-risk samples back to the old model. Practitioners can use these task-level patterns to choose which regression signal to trust for a given update. Code is available at https://github.com/jiashengsally/llm-regression-signals.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 34 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

28 of 34 references.

References · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
16 more
  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  2. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  3. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  4. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  5. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  6. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  7. 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 55
  8. 2024MUSCLE: A Model Update Strategy for Compatible LLM Evolutioncited by 15
  9. 2021Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updatescited by 13
  10. 2024What Will My Model Forget? Forecasting Forgotten Examples in Language Model Refinementcited by 9
  11. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  12. 2026Entropy Alone is Insufficient for Safe Selective Prediction in LLMscited by 7
  13. 2023Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusioncited by 6
  14. 2026Gradient-Stable Attention Heads Signal LLM Correctnesscited by 5
  15. 2026Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluationcited by 4
  16. 2025Mitigating Negative Flips via Margin Preserving Trainingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling
  2. 2025LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval
  3. 2026Predicting Inference-Time Scaling Gains from Labeled Validation-Set Output Statistics
  4. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
  5. 2026The Judge Knows When It Knows: Calibrated Abstention for LLM-Based A/B-Test Prediction
  6. 2026The Capability Frontier: Benchmarks Miss 82% of Model Performance
  7. 2025Efficient Test-Time Scaling via Self-Calibration
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs