mlsanityData to 14 Oct 2026

Training Language Models to Fail Gracefully

arXiv 2610.20401 ↗First posted 14 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning makes language models better at solving tasks, but the same mechanism also rewards guessing. When success is the only way to earn reward, opting for a safe fallback is always worse than a long-shot attempt. When a wrong action can corrupt a database or ship broken code, that incentive can be costly. Therefore, we propose training models to fail gracefully: to preserve successful behavior whenever possible while learning to execute a safe fallback policy when acting would be unreliable. However, naively using policy gradients with intermediate reward for fallback behavior is brittle due to unfavorable optimization dynamics. We address this challenge with GracefulRL, which reshapes the optimization landscape to promote graceful failure. Importantly, this behavior emerges directly from outcome-level reward signals, without annotations specifying when to fall back or additional inference-time machinery such as a verifier. Across a range of tasks and scales, we train models that largely preserve their task-solving capability while selectively falling back when acting would be unreliable. More broadly, our results show that reinforcement learning can not only incentivize models to succeed, but also to reason about the limits of their capabilities and change course when they determine that success is out of reach.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 54 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

35 of 54 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  8. 2017Constrained Policy Optimizationcited by 983
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  11. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  12. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
23 more
  1. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  2. 2018Understanding the impact of entropy on policy optimizationcited by 176
  3. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 175
  4. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  5. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  6. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
  7. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 111
  8. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Modelscited by 77
  9. 2019Deep Gamblers: Learning to Abstain with Portfolio Theorycited by 71
  10. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 66
  11. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 48
  12. 2026Maximum Likelihood Reinforcement Learningcited by 36
  13. 2025Training a Generally Curious Agentcited by 31
  14. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 27
  15. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  16. 2025Scaling Open-Ended Reasoning to Predict the Futurecited by 21
  17. 2025Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learningcited by 18
  18. 2026Abstain-R1: Calibrated Abstention and Post-Refusal Clarification via Verifiable RLcited by 14
  19. 2026Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoningcited by 8
  20. 2026GRACE: Reinforcement Learning for Grounded Response and Abstention under Contextual Evidencecited by 7
  21. 2026Delightful Policy Gradientcited by 3
  22. 2026TIAR: Trajectory-Informed Advantage Reweighting for LLM Abstention Learningcited by 3
  23. 2026Delightful Gradients Accelerate Corner Escapecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
  2. 2025The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models
  3. 2026SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents
  4. 2026EasyPPO: Stabilizing the Critic Is Key
  5. 2025Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective
  6. 2026Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It
  7. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworlds
  8. 2025Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL