mlsanityData to 14 Oct 2026

Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations

arXiv 2608.22138 ↗First posted 22 Aug 2026cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Language models are usually judged by a single accuracy score, which does not reveal how their performance degrades as inputs are perturbed. We present a graded, multi-family, failure-aware framework for stress-testing reasoning models. It perturbs each problem along a multi-level severity ladder across seven families: six that preserve the answer, paraphrase, input noise, formatting, irrelevant context, context load, and conflicting instructions, and a Knowledge Boundary family that removes answerability so that refusal becomes the correct response. Every test is validity-gated and labeled by its measured severity, and each model is summarized by per-level Accuracy, a magnitude-weighted Stability, and a per-family Collapse Point defined relative to the model's own baseline. Instantiated on the same 100 seed problems used by GSM-Symbolic, expanded into 4,473 gated tests and run on four models spanning capability tiers, the framework exposes structure that an aggregate score hides: the level at which a model fails is family-specific rather than global, and two stressors expose consistent weaknesses across all models: conflicting instructions and questions built on an impossible premise. Recognition of unanswerability is otherwise uneven, reliable on missing information and fabricated evidence but weak on impossible premises. These failure points are invisible to standard accuracy reporting.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 31 references that are in the map. First posted on arXiv 22 Aug 2026.

Builds on

29 of 31 references.

References · 29

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  8. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  9. 2022PAL: Program-aided Language Modelscited by 652
  10. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  12. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
17 more
  1. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  2. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  3. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  4. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  5. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  6. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  7. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 89
  8. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 72
  9. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Modelscited by 61
  10. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 48
  11. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Modelscited by 33
  12. 2023Revisit Input Perturbation Problems for LLMs: A Unified Robustness Evaluation Framework for Noisy Slot Filling Taskcited by 23
  13. 2025SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Modelscited by 19
  14. 2024RUPBench: Benchmarking Reasoning Under Perturbations for Robustness Evaluation in Large Language Modelscited by 13
  15. 2026MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviorscited by 4
  16. 2025Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Modelscited by 3
  17. 2026Robust Reasoning Benchmarkcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026"I May Not Have Articulated Myself Clearly": Diagnosing Dynamic Instability in LLM Reasoning at Inference Time
  2. 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations
  3. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
  4. 2026Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations
  5. 2026How Language Models Fail: Token-Level Signatures of Committed and Persistent Reasoning Failures
  6. 2023Evaluating Concurrent Robustness of Language Models Across Diverse Challenge Sets
  7. 2026Dissecting Failure Dynamics in Large Language Model Reasoning
  8. 2025FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation