mlsanityData to 14 Oct 2026

Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models

arXiv 2508.10751 ↗First posted 14 Aug 2025cs.LG cs.AI cs.CLcited by 138

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR), which typically adopts Pass@1 as the reward, has faced the issues in balancing exploration and exploitation, causing policies to prefer conservative actions, converging to a local optimum. Identifying an appropriate reward metric is therefore crucial. Regarding the prior work, although Pass@k has been used in evaluation, its connection to LLM exploration ability in RLVR remains largely overlooked. To investigate this, we first use Pass@k as the reward to train the policy model (i.e., Pass@k Training), and observe the improvement on its exploration ability. Next, we derive an analytical solution for the advantage of Pass@k Training, leading to an efficient and effective process. Building on this, our analysis reveals that exploration and exploitation are not inherently conflicting objectives, while they can mutually enhance each other. Moreover, Pass@k Training with analytical derivation essentially involves directly designing the advantage function. Inspired by this, we preliminarily explore the advantage design for RLVR, showing promising results and highlighting a potential future direction.

Key facts. Cited by 138 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 56 references that are in the map. First posted on arXiv 14 Aug 2025.

Builds on

46 of 56 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  3. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  4. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  5. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  7. 2025ARM: Adaptive Reasoning Model26 May
  8. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning21 May
  9. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers7 May
  10. 2025TTRL: Test-Time Reinforcement Learning22 Apr
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  12. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
11 more
  1. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  6. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning10 Feb
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  8. 2024OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning22 Dec
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 23

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  5. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  6. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  7. 2025Seed1.5-VL Technical Reportcited by 373
  8. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  9. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  10. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  11. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  12. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
11 more
  1. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 73
  2. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
  3. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  4. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 56
  5. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  6. 2025Test-Time Learning for Large Language Modelscited by 50
  7. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  8. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewardscited by 38
  9. 2025Learning from Peers in Reasoning Modelscited by 16
  10. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  11. 2025Thinker: Learning to Think Fast and Slowcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 51; 2026: 87.

138 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Outcome-based Exploration for LLM Reasoningcited by 83
  4. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  5. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 47
  6. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  7. 2026Maximum Likelihood Reinforcement Learningcited by 36
  8. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  9. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  10. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  11. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  12. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 22
38 more
  1. 2026Composer 2 Technical Reportcited by 22
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Modelscited by 21
  3. 2026Learning to Self-Verify Makes Language Models Better Reasonerscited by 21
  4. 2025Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoningcited by 17
  5. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  6. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  7. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 15
  8. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 14
  9. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forgetcited by 14
  10. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  11. 2025Reinforcement Learning on Pre-Training Datacited by 12
  12. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  13. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Modelscited by 12
  14. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Trainingcited by 12
  15. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 12
  16. 2025Pass@k Metric for RLVR: A Diagnostic Tool of Exploration, But Not an Objectivecited by 12
  17. 2026CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMscited by 12
  18. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
  19. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradientscited by 11
  20. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoningcited by 11
  21. 2026Entropy-Preserving Reinforcement Learningcited by 11
  22. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewardscited by 10
  23. 2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learningcited by 10
  24. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 10
  25. 2025MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Modelcited by 10
  26. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 10
  27. 2026On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversitycited by 10
  28. 2025Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scalingcited by 9
  29. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  30. 2025Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundariescited by 9
  31. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  32. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  33. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 9
  34. 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversitycited by 9
  35. 2025Exploring Multi-Temperature Strategies for Token- and Rollout-Level Control in RLVRcited by 8
  36. 2025Mode-Conditioning Unlocks Superior Test-Time Scalingcited by 8
  37. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 8
  38. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  2. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  3. 2026Decoupling Exploration from Optimization in RLVR
  4. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  5. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  6. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR
  7. 2026Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance
  8. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward