mlsanityData to 7 Oct 2026

First Return, Entropy-Eliciting Explore

arXiv 2507.07017 ↗First posted 9 Jul 2025cs.AIcited by 45

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning from Verifiable Rewards (RLVR) improves the reasoning abilities of Large Language Models (LLMs) but it struggles with unstable exploration. We propose FR3E (First Return, Entropy-Eliciting Explore), a structured exploration framework that identifies high-uncertainty decision points in reasoning trajectories and performs targeted rollouts to construct semantically grounded intermediate feedback. Our method provides targeted guidance without relying on dense supervision. Empirical results on mathematical reasoning benchmarks(AIME24) show that FR3E promotes more stable training, produces longer and more coherent responses, and increases the proportion of fully correct trajectories. These results highlight the framework's effectiveness in improving LLM reasoning through more robust and structured exploration.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 33 references that are in the map. First posted on arXiv 9 Jul 2025.

Builds on

27 of 33 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  4. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  9. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  8. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  9. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  10. 2020First return, then explorecited by 256
  11. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  12. 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
5 more
  1. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  2. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  3. 2024Intelligent Go-Explore: Standing on the Shoulders of Giant Foundation Modelscited by 32
  4. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  5. 2025Temporal Sampling for Forgotten Reasoning in LLMscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 22.

45 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 285
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025Agentic Reinforced Policy Optimizationcited by 166
  4. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  5. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  6. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 32
  7. 2025Rethinking Entropy Regularization in Large Reasoning Modelscited by 25
  8. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
  9. 2025Semantic Energy: Detecting LLM Hallucination Beyond Entropycited by 18
  10. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 18
  11. 2025Attention as a Compass: Efficient Exploration for Process-Supervised RL in Reasoning Modelscited by 13
  12. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
33 more
  1. 2025Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learningcited by 12
  2. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 11
  3. 2025Bootstrapping Task Spaces for Self-Improvementcited by 10
  4. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  5. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMscited by 10
  6. 2026Reinforced Efficient Reasoning via Semantically Diverse Explorationcited by 8
  7. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Modelscited by 7
  8. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  9. 2025ReLaX: Reasoning with Latent Exploration for Large Reasoning Modelscited by 4
  10. 2025Self-Reflective Generation at Test Timecited by 3
  11. 2025Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learningcited by 3
  12. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 3
  13. 2026From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoningcited by 3
  14. 2025WeChat-YATT: A Scalable, Simple, Efficient, and Production Ready Training Librarycited by 2
  15. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  16. 2026R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learningcited by 2
  17. 2026LAD: Learning Advantage Distribution for Reasoningcited by 2
  18. 2026The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Creditscited by 2
  19. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priorscited by 2
  20. 2026Reasoning Can Be Restored by Correcting a Few Decision Tokenscited by 2
  21. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2
  22. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  23. 2026Optimizing What Policies Learn From: Recoverability-aware Rollout Intervention Learningcited by 2
  24. 2025Spectral Logit Sculpting: Adaptive Low-Rank Logit Transformation for Controlled Text Generationcited by 1
  25. 2026Learning from Self-Debate: Preparing Reasoning Models for Multi-Agent Debatecited by 1
  26. 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMscited by 1
  27. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 1
  28. 2026Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carriercited by 1
  29. 2026Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimizationcited by 1
  30. 2026Deep Dense Exploration for LLM Reinforcement Learning via Pivot-Driven Resamplingcited by 0
  31. 2026What are Key Factors for Updates in RL for LLM Reasoning?cited by 0
  32. 2026Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learningcited by 0
  33. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning with Exploration: An Entropy Perspective
  2. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  3. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  4. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning
  5. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  6. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  7. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration
  8. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning