mlsanityData to 14 Oct 2026

CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention

arXiv 2508.11016 ↗First posted 14 Aug 2025cs.LG cs.AIcited by 18

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent advances in Reinforcement Learning with Verified Reward (RLVR) have driven the emergence of more sophisticated cognitive behaviors in large language models (LLMs), thereby enhancing their reasoning capabilities. However, in prior RLVR pipelines, the repeated use of static initial-state sampling drawn exactly from the dataset distribution during each sampling phase produced overly deterministic, low diversity model behavior, which manifested as rapid entropy collapse and hindered sustained performance gains during prolonged training. To address this issue, we introduce CURE (Critical-token-gUided Re concatenation for Entropy-collapse prevention), a two-stage framework that balances exploration and exploitation. Specifically, in the first stage, to deliberately steer the model toward novel yet coherent contexts, we re-generate at high-entropy critical tokens and jointly optimize the original and the branched trajectories. The further comparison with vanilla DAPO shows that the regeneration process achieves a better performance on math reasoning tasks while sustaining a high-level entropy degree for exploration. In the second stage, we continue training with static initial-state sampling by DAPO, intentionally placing the model in a familiar state to gradually strengthen exploitation. Extensive experiments on Qwen-2.5-Math-7B show that, compared to other RLVR methods, CURE achieves a 5% performance gain across six math benchmarks, establishing state-of-the-art performance in both entropy and accuracy. A series of experiments further validate the effectiveness of our approach. Code is available at https://github.com/bytedance/CURE.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 32 references that are in the map. First posted on arXiv 14 Aug 2025.

Builds on

24 of 32 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoning23 May
  5. 2025Learning to Reason under Off-Policy Guidance21 Apr
  6. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  7. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  9. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  10. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  11. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  12. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 165
1 more
  1. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 10.

18 papers.

Sort:

By area:

  1. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  2. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 52
  3. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  4. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Modelscited by 21
  5. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learningcited by 11
  6. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  7. 2025Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Modelscited by 6
  8. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  9. 2025ReLaX: Reasoning with Latent Exploration for Large Reasoning Modelscited by 5
  10. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  11. 2026Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoningcited by 4
  12. 2025Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generationcited by 2
6 more
  1. 2026Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMscited by 1
  2. 2026Boosting LLM Reasoning via Human-Inspired Reward Shapingcited by 1
  3. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoningcited by 1
  4. 2026Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understandingcited by 0
  5. 2026GRPODropout: Less is More for Online Reinforcement Learning Rolloutscited by 0
  6. 2026Branching On-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  2. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
  3. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  4. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
  5. 2026Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization
  6. 2026HEALing Entropy Collapse: Enhancing Exploration in Few-Shot RLVR via Hybrid-Domain Entropy Dynamics Alignment
  7. 2026Depth-Entropy Guided Sampling for Training-Free LLM Reasoning
  8. 2026Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective