mlsanityData to 14 Oct 2026

Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration

arXiv 2505.17621 ↗First posted 23 May 2025cs.LGcited by 34

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning (RL) has become a key approach for enhancing the reasoning capabilities of large language models. However, prevalent RL approaches like proximal policy optimization and group relative policy optimization suffer from sparse, outcome-based rewards and weak exploration incentives, limiting their effectiveness. Specifically, sparse rewards offer limited feedback, especially on difficult problems, and introduce biases favoring familiar trajectories over novel reasoning paths. These issues critically undermine performance on complex tasks that inherently require iterative reasoning. To overcome these challenges, we propose Intrinsic MotivAtion Guided exploratIoN for Enhanced reasoning (IMAGINE), which delivers dense rewards and encourages exploration. IMAGINE introduces three innovations: a trajectory-aware exploration reward that reduces token-level bias efficiently; an error-conditioned reward allocation that promotes efficient exploration on hard samples while stabilizing training; and an advantage-preserving integration mechanism that retains distributional integrity during learning. Experiments on four public datasets show that IMAGINE improves performance by 22.23% on AIME 2024.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 52 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

37 of 52 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Improving RL Exploration for LLM Reasoning through Retrospective Replay19 Apr
  2. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  6. 2023Let's Verify Step by Step31 May
  7. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  10. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  11. 2018Exploration by Random Network Distillationcited by 993
  12. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
17 more
  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  2. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  3. 2017Count-Based Exploration with Neural Density Modelscited by 257
  4. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  5. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 196
  6. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  7. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  8. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  9. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  10. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  11. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 99
  12. 2024Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Surveycited by 83
  13. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  14. 2025Steering Large Language Model Activations in Sparse Spacescited by 58
  15. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  16. 2024Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generationcited by 19
  17. 2024Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 21.

34 papers.

Sort:

By area:

  1. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  2. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoningcited by 84
  3. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  4. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  5. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 35
  6. 2025CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Modelscited by 29
  7. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 17
  8. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  9. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  10. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 14
  11. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Modelscited by 12
  12. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 12
22 more
  1. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learningcited by 11
  2. 2025Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewardscited by 9
  3. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 9
  4. 2025Dual-Uncertainty Guided Policy Learning for Multimodal Reasoningcited by 5
  5. 2026MemSearch-o1: Empowering Large Language Models with Reasoning-Aligned Memory Growth in Agentic Searchcited by 5
  6. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 5
  7. 2025BaNEL: Exploration Posteriors for Generative Modeling Using Only Negative Rewardscited by 3
  8. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimizationcited by 3
  9. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 3
  10. 2026Tandem: Riding Together with Large and Small Language Models for Efficient Reasoningcited by 2
  11. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priorscited by 2
  12. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  13. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 2
  14. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  15. 2025Why Did Apple Fall: Evaluating Curiosity in Large Language Modelscited by 0
  16. 2026BRIDGE: Bridging Reasoning In Distillation Gap Elimination via Structure-Aware Maskingcited by 0
  17. 2026AdapTime: Enabling Adaptive Temporal Reasoning in Large Language Modelscited by 0
  18. 2026MultiDx: A Multi-Source Knowledge Integration Framework towards Diagnostic Reasoningcited by 0
  19. 2026Epistemic Uncertainty for Test-Time Discoverycited by 0
  20. 2026ExpRL: Exploratory RL for LLM Mid-Trainingcited by 0
  21. 2026CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discoverycited by 0
  22. 2026Decoupling Exploration from Optimization in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
  2. 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs
  3. 2025Know When to Explore: Difficulty-Aware Certainty as a Guide for LLM Reinforcement Learning
  4. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  5. 2025Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
  6. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  7. 2025Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models
  8. 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning