mlsanityData to 7 Oct 2026

Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models

arXiv 2509.24261 ↗First posted 29 Sep 2025cs.AI cs.LGcited by 12

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for enhancing Large Language Models (LLMs) on complex reasoning tasks. However, existing methods suffer from an exploration dilemma: the sharply peaked initial policies of pre-trained LLMs confine standard RL algorithms to a narrow set of solutions, boosting single-solution accuracy (pass@1) but suppressing solution diversity and multi-solution performance (pass@k). As a result, RLVR often distills existing capabilities rather than discovering new reasoning strategies. To overcome this, we introduce a Risk-Sensitive Reinforcement Learning framework. Our approach employs a risk-seeking objective that interpolates between mean and maximum rewards, leading to a novel algorithm, Risk-Sensitive GRPO (RS-GRPO), which drives deeper exploration by amplifying learning from challenging prompts. Remarkably, RS-GRPO is simple to implement, requiring only minor code modifications. On six mathematical reasoning benchmarks and with five different LLMs, RS-GRPO consistently improves pass@k performance while maintaining or enhancing pass@1 accuracy.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 51 of its 63 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

51 of 63 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  2. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  3. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  4. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  5. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  7. 2025Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration23 May
  8. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  9. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
1 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan

Other references · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2017A Distributional Perspective on Reinforcement Learningcited by 743
26 more
  1. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  2. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  3. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  4. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 323
  5. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  6. 2019SPoC: Search-based Pseudocode to Codecited by 176
  7. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  8. 2013Risk-sensitive Reinforcement Learningcited by 129
  9. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  10. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  11. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  12. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 59
  13. 2022Exploration via Elliptical Episodic Bonusescited by 51
  14. 2024Variational Best-of-N Alignmentcited by 49
  15. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
  16. 2025Rethinking Reflection in Pre-Trainingcited by 43
  17. 2024InfAlign: Inference-aware language model alignmentcited by 34
  18. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  19. 2025PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifiercited by 28
  20. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 24
  21. 2019Entropic Risk Measure in Policy Searchcited by 19
  22. 2023Robust Quadrupedal Locomotion via Risk-Averse Policy Learningcited by 16
  23. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  24. 2025Episodic Novelty Through Temporal Distancecited by 15
  25. 2019Reinforcement Learning for Portfolio Managementcited by 12
  26. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  2. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 6
  3. 2026PACEvolve++: Improving Test-time Learning for Evolutionary Search Agentscited by 3
  4. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPOcited by 1
  5. 2026Boosting LLM Reasoning via Human-Inspired Reward Shapingcited by 1
  6. 2026All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Modelscited by 1
  7. 2026Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPscited by 1
  8. 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoningcited by 1
  9. 2026Reinforcement Learning without Ground-Truth Solutions can Improve LLMscited by 0
  10. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  11. 2026Planned Test-Time Scaling with Coordinated Reasoning Pathscited by 0
  12. 2026Exploring More, Reasoning Better: Stepwise Risk-Sensitive GRPO for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
  2. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifiers
  3. 2025Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
  4. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
  5. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  6. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  7. 2025Unlocking Exploration in RLVR: Uncertainty-aware Advantage Shaping for Deeper Reasoning
  8. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning