mlsanityData to 7 Oct 2026

The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models

arXiv 2507.20150 ↗First posted 27 Jul 2025cs.AI cs.CL cs.LGcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) plays a crucial role in shaping the behavior of large language and reasoning models (LLMs/LRMs). However, it often produces brittle and unstable policies, leading to critical failures such as spurious reasoning, deceptive alignment, and instruction disobedience that undermine the trustworthiness and safety of LLMs/LRMs. Currently, these issues lack a unified theoretical explanation and are typically addressed using ad-hoc heuristics. This paper presents a rigorous mathematical framework for analyzing the stability of the mapping from a reward function to the optimal policy. We show that policy brittleness often stems from non-unique optimal actions, a common occurrence when multiple valid traces exist in a reasoning task. This theoretical lens provides a unified explanation for a range of seemingly disparate failures, reframing them as rational outcomes of optimizing rewards that may be incomplete or noisy, especially in the presence of action degeneracy. We extend this analysis from the fundamental single-reward setting to the more realistic multi-reward RL across diverse domains, showing how stability is governed by an "effective reward" aggregation mechanism. We also prove that entropy regularization restores policy stability at the cost of increased stochasticity. Our framework provides a unified explanation for recent empirical findings on deceptive reasoning, instruction-following trade-offs, and RLHF-induced sophistry, and is further validated through perturbation experiments in multi-reward RL. This work advances policy-stability analysis from empirical heuristics towards a principled theory, offering essential insights for designing safer and more trustworthy AI systems.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 44 references that are in the map. First posted on arXiv 27 Jul 2025.

Builds on

28 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
  2. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  3. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains31 Mar
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  7. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  8. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  9. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  10. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  11. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  12. 2019A Theory of Regularized Markov Decision Processescited by 279
10 more
  1. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  2. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  3. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  4. 2018Reinforcement Learning with Perturbed Rewardscited by 76
  5. 2025Persona Features Control Emergent Misalignmentcited by 76
  6. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  7. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  8. 2020Lipschitz Lifelong Reinforcement Learningcited by 23
  9. 2021Defense Against Reward Poisoning Attacks in Reinforcement Learningcited by 20
  10. 2023Behavior Alignment via Reward Function Optimizationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generationcited by 3
  2. 2026PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficientcited by 0
  3. 2026Epistemic Traps: Rational Misalignment Driven by Model Misspecificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
  2. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  3. 2026Cliff: Learning Process Rewards from the First Mistake
  4. 2026Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
  5. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Making
  6. 2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
  7. 2026Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
  8. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models