mlsanityData to 14 Oct 2026

Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMs

arXiv 2606.17735 ↗First posted 16 Jun 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Although reinforcement learning (RL) has expanded the cognitive boundaries of large language models (LLMs), it often remains vulnerable to the autoregressive curse in long-horizon logical reasoning: small epistemic perturbations introduced early in generation can propagate irreversibly along the Markov decision process flow, triggering cascading failures that drive the reasoning trajectory toward collapse. To overcome this autoregressive cascade, in which a single early mistake can compromise all subsequent reasoning steps, we propose dynamic epistemic entropy orchestrated erasable reinforcement learning (E3RL). E3RL eliminates reliance on external signals by grounding the model's endogenous local autoregressive cross-entropy as an intrinsic coordinate of epistemic uncertainty. By introducing segment-level adaptive dynamic thresholds and advantage allocation, E3RL enables the model to precisely excise localized logical defects while reusing historical key-value (KV) cache streams, thereby endowing the reasoning process with a self-healing capability. We train E3RL on the DeepMath-103k dataset. Experimental results show that E3RL reshapes the exploration efficiency of long-sequence reasoning and improves sample efficiency while maintaining linear memory overhead. On mathematical reasoning benchmarks such as AIME, E3RL achieves substantial performance gains, with the 4B and 8B parameter models surpassing previous state-of-the-art (SOTA) results by 5.349% and 6.514%, respectively. These findings suggest that E3RL shatters the autoregressive curse in long-sequence reasoning and establishes a theoretical and systems-level foundation for the next generation of self-healing artificial general intelligence (AGI).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 63 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

60 of 63 references.

Closest ancestors · 19

Same direction, newest first

  1. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism6 Apr
  2. 2026PRISM: Pushing the Frontier of Deep Think via Process Reward Model-Guided Inference3 Mar
  3. 2026Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models20 Feb
  4. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement31 Jan
  5. 2026Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning23 Jan
  6. 2026InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning20 Jan
  7. 2025Soft Adaptive Policy Optimization25 Nov
  8. 2025A Survey of Process Reward Models: From Outcome Signals to Process Supervisions for Large Language Models9 Oct
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  10. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
  11. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning10 Jun
  12. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
7 more
  1. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models29 May
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  3. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization18 May
  4. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoning20 Dec
  7. 2023Let's Verify Step by Step31 May

Other references · 41

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  5. 2025Group Sequence Policy Optimizationcited by 835
  6. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  7. 2025ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learningcited by 192
  8. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 108
  9. 2025StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimizationcited by 88
  10. 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 72
  11. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 57
  12. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
29 more
  1. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  2. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  3. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  4. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  5. 2026Large Language Model Reasoning Failurescited by 21
  6. 2026Proximity-Based Multi-Turn Optimization: Practical Credit Assignment for LLM Agent Trainingcited by 18
  7. 2025Probabilistic Soundness Guarantees in LLM Reasoning Chainscited by 13
  8. 2026V1: Unifying Generation and Self-Verification for Parallel Reasonerscited by 12
  9. 2026EDIS: Diagnosing LLM Reasoning via Entropy Dynamicscited by 10
  10. 2026HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agentscited by 9
  11. 2025Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMscited by 6
  12. 2026IQuest-Coder-V1 Technical Reportcited by 6
  13. 2026DiffCoT: Diffusion-styled Chain-of-Thought Reasoning in LLMscited by 5
  14. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  15. 2024YuLan: An Open-source Large Language Modelcited by 4
  16. 2026InCoder-32B: Code Foundation Model for Industrial Scenarioscited by 4
  17. 2026Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Modelscited by 4
  18. 2026ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoningcited by 4
  19. 2026Unveiling the Entropy Dynamics of Chain-of-Thought Reasoningcited by 4
  20. 2025Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibriumcited by 3
  21. 2026CoRefine: Confidence-Guided Self-Refinement for Adaptive Test-Time Computecited by 3
  22. 2026Learning Adaptive LLM Decodingcited by 3
  23. 2026More Test-Time Compute Can Hurt: Overestimation Bias in LLM Beam Searchcited by 3
  24. 2026Flexible Empowerment at Reasoning with Extended Best-of-N Samplingcited by 3
  25. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 3
  26. 2026PILOT: Planning via Internalized Latent Optimization Trajectories for Large Language Modelscited by 2
  27. 2026Reasoning Can Be Restored by Correcting a Few Decision Tokenscited by 2
  28. 2025MMRPT: MultiModal Reinforcement Pre-Training via Masked Vision-Dependent Reasoningcited by 1
  29. 2026Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  3. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  4. 2025Rethinking Entropy Regularization in Large Reasoning Models
  5. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  6. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
  7. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  8. 2025Reasoning with Exploration: An Entropy Perspective