mlsanityData to 7 Oct 2026

Reasoning with Exploration: An Entropy Perspective

arXiv 2506.14758 ↗First posted 17 Jun 2025cs.CLcited by 277

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Balancing exploration and exploitation is a central goal in reinforcement learning (RL). Despite recent advances in enhancing large language model (LLM) reasoning, most methods lean toward exploitation, and increasingly encounter performance plateaus. In this work, we revisit entropy -- a signal of exploration in RL -- and examine its relationship to exploratory reasoning in LLMs. Through empirical analysis, we uncover positive correlations between high-entropy regions and three types of exploratory reasoning actions: (1) pivotal tokens that determine or connect logical steps, (2) reflective actions such as self-verification and correction, and (3) rare behaviors under-explored by the base LLMs. Motivated by this, we introduce a minimal modification to standard RL with only one line of code: augmenting the advantage function with an entropy-based term. Unlike traditional maximum-entropy methods which encourage exploration by promoting uncertainty, we encourage exploration by promoting longer and deeper reasoning chains. Notably, our method achieves significant gains on the Pass@K metric -- an upper-bound estimator of LLM reasoning capabilities -- even when evaluated with extremely large K values, pushing the boundaries of LLM reasoning.

Key facts. Cited by 277 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 53 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

47 of 53 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration23 May
  5. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  6. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  8. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  10. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  11. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  12. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
8 more
  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  5. 2025Process Reinforcement through Implicit Rewards3 Feb
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
15 more
  1. 2018Exploration by Random Network Distillationcited by 966
  2. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  3. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  4. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  5. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
  6. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  7. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  8. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  9. 2022Exploration via Elliptical Episodic Bonusescited by 51
  10. 2024Self-Consistency Preference Optimizationcited by 50
  11. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 50
  12. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  13. 2025Rethinking Reflection in Pre-Trainingcited by 43
  14. 2016Combining policy gradient and Q-learningcited by 23
  15. 2025CAE: Repurposing the Critic as an Explorer in Deep Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 115; 2026: 162.

277 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025Agentic Reinforced Policy Optimizationcited by 166
  4. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  5. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 134
  6. 2026Self-Distilled RLVRcited by 109
  7. 2025Geometric-Mean Policy Optimizationcited by 107
  8. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  9. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  10. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  11. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  12. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 53
38 more
  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 50
  2. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 48
  3. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  4. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  5. 2025Emergent Hierarchical Reasoning in LLMs through Reinforcement Learningcited by 42
  6. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 42
  7. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversitycited by 41
  8. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  9. 2025Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVRcited by 38
  10. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  11. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reasoncited by 37
  12. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  13. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 36
  14. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
  15. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  16. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoningcited by 34
  17. 2025Guided Self-Evolving LLMs with Minimal Human Supervisioncited by 33
  18. 2025Pruning the Unsurprising: Efficient LLM Reasoning via First-Token Surprisalcited by 32
  19. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 32
  20. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 32
  21. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 29
  22. 2026Maximum Likelihood Reinforcement Learningcited by 29
  23. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  24. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  25. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  26. 2025Rethinking Entropy Regularization in Large Reasoning Modelscited by 25
  27. 2025Nudging the Boundaries of LLM Reasoningcited by 25
  28. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
  29. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
  30. 2025RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agentscited by 24
  31. 2025Entropy After </Think> for reasoning model early exitingcited by 24
  32. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learningcited by 23
  33. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  34. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learningcited by 22
  35. 2025Revisiting LLM Reasoning via Information Bottleneckcited by 19
  36. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 19
  37. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  38. 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropycited by 18

50 most cited of 277 All 277 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025First Return, Entropy-Eliciting Explore
  2. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  3. 2026Beyond Entropy: Learning from Token-Level Distributional Deviations for LLM Reasoning
  4. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors
  5. 2025Explore Briefly, Then Decide: Mitigating LLM Overthinking via Cumulative Entropy Regulation
  6. 2026Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control
  7. 2025Entropy-based Exploration Conduction for Multi-step Reasoning
  8. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning