mlsanityData to 7 Oct 2026

Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward

arXiv 2512.16912 ↗First posted 18 Dec 2025cs.LG cs.AI cs.CLcited by 25

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

This paper examines the exploration-exploitation trade-off in reinforcement learning with verifiable rewards (RLVR), a framework for improving the reasoning of Large Language Models (LLMs). Recent studies suggest that RLVR can elicit strong mathematical reasoning in LLMs through two seemingly paradoxical mechanisms: spurious rewards, which suppress exploitation by rewarding outcomes unrelated to the ground truth, and entropy minimization, which suppresses exploration by pushing the model toward more confident and deterministic outputs, highlighting a puzzling dynamic: both discouraging exploitation and discouraging exploration improve reasoning performance, yet the underlying principles that reconcile these effects remain poorly understood. We focus on two fundamental questions: (i) how policy entropy relates to performance, and (ii) whether spurious rewards yield gains, potentially through the interplay of clipping bias and model contamination. Our results show that clipping bias under spurious rewards reduces policy entropy, leading to more confident and deterministic outputs, while entropy minimization alone is insufficient for improvement. We further propose a reward-misalignment model explaining why spurious rewards can enhance performance beyond contaminated settings. Our findings clarify the mechanisms behind spurious-reward benefits and provide principles for more effective RLVR training.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 74 of its 93 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

74 of 93 references.

Closest ancestors · 28

Same direction, newest first

  1. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  2. 2025The Majority is not always right: RL training for solution aggregation8 Sep
  3. 2025Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination14 Jul
  4. 2025First Return, Entropy-Eliciting Explore9 Jul
  5. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  6. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  7. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  8. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning29 May
  9. 2025Maximizing Confidence Alone Improves Reasoning28 May
  10. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  11. 2025Learning to Reason without External Rewards26 May
  12. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
16 more
  1. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO16 May
  2. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  3. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  7. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  8. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  9. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervision5 Jun
  10. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  11. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  12. 2023Let's Verify Step by Step31 May
  13. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  14. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  15. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  16. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 46

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  11. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
34 more
  1. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  3. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  5. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  6. 2013Learning to Optimize Via Posterior Samplingcited by 253
  7. 2024Iterative Reasoning Preference Optimizationcited by 244
  8. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  9. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
  10. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 214
  11. 2025Deep Think with Confidencecited by 199
  12. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  13. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  14. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  15. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  16. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  17. 2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 108
  18. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  19. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  20. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  21. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learningcited by 80
  22. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  23. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  24. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  25. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 52
  26. 2025Self-Questioning Language Modelscited by 47
  27. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  28. 2025On Entropy Control in LLM-RL Algorithmscited by 39
  29. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32
  30. 2025One-shot Entropy Minimizationcited by 32
  31. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedbackcited by 14
  32. 2025ComPO: Preference Alignment via Comparison Oraclescited by 11
  33. 2025Consensus-Driven Active Model Selectioncited by 7
  34. 2025Exploration by Random Reward Perturbationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 24.

25 papers.

Sort:

By area:

  1. 2025GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulatorscited by 34
  2. 2026Maximum Likelihood Reinforcement Learningcited by 29
  3. 2026Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMscited by 13
  4. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 11
  5. 2026Reward-free Alignment for Conflicting Objectivescited by 10
  6. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  7. 2026Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoningcited by 5
  8. 2026CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVRcited by 5
  9. 2026StaRPO: Stability-Augmented Reinforcement Policy Optimizationcited by 5
  10. 2026OmniOPD: Logit-Free On-Policy Distillation via Speculative Verificationcited by 5
  11. 2026Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Trainingcited by 4
  12. 2026VeriGate: Verifier-Gated Step-Level Supervision for GRPOcited by 3
13 more
  1. 2026Policy Improvement Reinforcement Learningcited by 2
  2. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Controlcited by 2
  3. 2026PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVRcited by 2
  4. 2026When Can LLMs Learn to Reason with Weak Supervision?cited by 1
  5. 2026Two-Fidelity Best-Action Identification for Stochastic Minimax Treecited by 1
  6. 2026Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspectivecited by 0
  7. 2026Two is better than one: A Collapse-free Multi-Reward RLIF Training Frameworkcited by 0
  8. 2026EAPO: Entropy-Driven Adaptive Positive-Negative Sample Weighting for Policy Optimization in Open-Ended QAcited by 0
  9. 2026Parameter Exploration for RLVR via Variational Learningcited by 0
  10. 2026A Zeroth-Order Paradigm for LLM Preference Alignmentcited by 0
  11. 2026Towards Full Pipeline FP8 Reinforcement Learning for LLMscited by 0
  12. 2026A Cheap Verifier is Good Enough: LLM Post-training is Robust to Erroneous Rewardscited by 0
  13. 2026Learned Reporting Preferences in RLVR Can Conflict with the Current Requestcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  2. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  3. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  4. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
  5. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning
  6. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models
  7. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  8. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs