mlsanityData to 14 Oct 2026

CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models

arXiv 2509.09675 ↗First posted 11 Sep 2025cs.CL cs.AI cs.LGcited by 29

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful paradigm for enhancing the reasoning ability of Large Language Models (LLMs). Yet current RLVR methods often explore poorly, leading to premature convergence and entropy collapse. To address this challenge, we introduce Curiosity-Driven Exploration (CDE), a framework that leverages the model's own intrinsic sense of curiosity to guide exploration. We formalize curiosity with signals from both the actor and the critic: for the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture. Both signals serve as an exploration bonus within the RLVR framework to guide the model. Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses; moreover, we connect the critic-wise bonus to the well-established count-based exploration bonus in RL. Empirically, our method achieves an approximate +3 point improvement over standard RLVR using GRPO/PPO on AIME benchmarks. Further analysis identifies a calibration collapse mechanism within RLVR, shedding light on common LLM failure modes.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 60 references that are in the map. First posted on arXiv 11 Sep 2025.

Builds on

50 of 60 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  2. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration23 May
  3. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  7. 2024AlphaMath Almost Zero: Process Supervision without Process6 May
  8. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 41

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  9. 2015Continuous control with deep reinforcement learningcited by 3,407
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
29 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  2. 2018Exploration by Random Network Distillationcited by 993
  3. 2016Deep Exploration via Bootstrapped DQNcited by 527
  4. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  5. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
  6. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  7. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  8. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  9. 2025Deep Think with Confidencecited by 205
  10. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  11. 2024Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planningcited by 97
  12. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
  13. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  14. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 73
  15. 2024Calibrating Large Language Models Using Their Generations Onlycited by 65
  16. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 61
  17. 2025One Token to Fool LLM-as-a-Judgecited by 60
  18. 2025On Entropy Control in LLM-RL Algorithmscited by 40
  19. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  20. 2022Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximationcited by 31
  21. 2024LiteSearch: Efficacious Tree Search for LLMcited by 31
  22. 2024Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learningcited by 30
  23. 2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
  24. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 26
  25. 2025Learning to Reason via Mixture-of-Thought for Logical Reasoningcited by 20
  26. 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Studycited by 17
  27. 2025R1-RE: Cross-Domain Relation Extraction with RLVRcited by 11
  28. 2024Warm-up Free Policy Optimization: Improved Regret in Linear Markov Decision Processescited by 5
  29. 2025Breach in the Shield: Unveiling the Vulnerabilities of Large Language Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 20.

29 papers.

Sort:

By area:

  1. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 47
  2. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  3. 2025Does Tone Change the Answer? Evaluating Prompt Politeness Effects on Modern LLMs: GPT, Gemini, and LLaMAcited by 23
  4. 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 21
  5. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 15
  6. 2026CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMscited by 12
  7. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learningcited by 11
  8. 2025Stable and Efficient Single-Rollout RL for Multimodal Reasoningcited by 8
  9. 2026RelayLLM: Efficient Reasoning via Collaborative Decodingcited by 8
  10. 2025StatEval: A Comprehensive Benchmark for Large Language Models in Statisticscited by 6
  11. 2025Dual-Uncertainty Guided Policy Learning for Multimodal Reasoningcited by 5
  12. 2026Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Modelscited by 5
17 more
  1. 2026Calibration-Aware Policy Optimization for Reasoning LLMscited by 5
  2. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 5
  3. 2025ICPO: Intrinsic Confidence-Driven Group Relative Preference Optimization for Efficient Reinforcement Learningcited by 4
  4. 2026Distribution-Aware Reward Estimation for Test-Time Reinforcement Learningcited by 3
  5. 2026Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMscited by 3
  6. 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoningcited by 3
  7. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verificationcited by 3
  8. 2026Tail-Likelihood Reinforcement Learningcited by 3
  9. 2026Policy Improvement Reinforcement Learningcited by 2
  10. 2026Reinforcing Multimodal Reasoning Against Visual Degradationcited by 2
  11. 2025UniRel: Relation-Centric Knowledge Graph Question Answering with RL-Tuned LLM Reasoningcited by 1
  12. 2026Curiosity Driven Knowledge Retrieval for Mobile Agentscited by 1
  13. 2026DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Offcited by 0
  14. 2026Epistemic Uncertainty for Test-Time Discoverycited by 0
  15. 2026SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Mattercited by 0
  16. 2026Align Then Reason: A Multimodal Lip-Sync Judge for Dubbingcited by 0
  17. 2026VICO: Visual Environments Co-Evolving for Vision-Language Model Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  2. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  3. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
  4. 2026Decoupling Exploration from Optimization in RLVR
  5. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  6. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  7. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priors
  8. 2025Curiosity-Driven Reinforcement Learning from Human Feedback