mlsanityData to 7 Oct 2026

Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards

arXiv 2512.21625 ↗First posted 25 Dec 2025cs.CLcited by 19

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large reasoning models (LRMs) are typically trained using reinforcement learning with verifiable reward (RLVR) to enhance their reasoning abilities. In this paradigm, policies are updated using both positive and negative self-generated rollouts, which correspond to distinct sample polarities. In this paper, we provide a systematic investigation into how these sample polarities affect RLVR training dynamics and behaviors. We find that positive samples sharpen existing correct reasoning patterns, while negative samples encourage exploration of new reasoning paths. We further explore how adjusting the advantage values of positive and negative samples at both the sample level and the token level affects RLVR training. Based on these insights, we propose an Adaptive and Asymmetric token-level Advantage shaping method for Policy Optimization, namely A3PO, that more precisely allocates advantage signals to key tokens across different polarities. Experiments across five reasoning benchmarks demonstrate the effectiveness of our approach.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 42 references that are in the map. First posted on arXiv 25 Dec 2025.

Builds on

26 of 42 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  2. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward3 Oct
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  4. 2025Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward1 Sep
  5. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  6. 2025Geometric-Mean Policy Optimization28 Jul
  7. 2025Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study5 Jun
  8. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  9. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  10. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  11. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  12. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
1 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 13

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  8. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  9. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  10. 2025Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineeringcited by 34
  11. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspectivecited by 12
  12. 2024Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillationcited by 5
1 more
  1. 2025L2V-CoT: Cross-Modal Transfer of Chain-of-Thought Reasoning via Latent Interventioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 19.

19 papers.

Sort:

By area:

  1. 2026Exploring Reasoning Reward Model for Agentscited by 13
  2. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 9
  3. 2026Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVRcited by 8
  4. 2026Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learningcited by 6
  5. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
  6. 2026Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learningcited by 3
  7. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  8. 2026Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Indexcited by 3
  9. 2026Perceive-to-Reason: Decoupling Perception and Reasoning for Fine-Grained Visual Reasoningcited by 3
  10. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 2
  11. 2026The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Creditscited by 2
  12. 2026Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Controlcited by 2
7 more
  1. 2026CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditionscited by 0
  2. 2026MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learningcited by 0
  3. 2026EAPO: Entropy-Driven Adaptive Positive-Negative Sample Weighting for Policy Optimization in Open-Ended QAcited by 0
  4. 2026RL Post-Training Builds Compositional Reasoning Strategiescited by 0
  5. 2026Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoningcited by 0
  6. 2026Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratificationcited by 0
  7. 2026Surprising Success, Repeated Failure: Entropy-Guided Credit Assignment for Exploration in LLM Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  3. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  4. 2025Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models
  5. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
  6. 2025TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning
  7. 2026Rewards as Labels: Revisiting RLVR from a Classification Perspective
  8. 2026What are Key Factors for Updates in RL for LLM Reasoning?