mlsanityData to 7 Oct 2026

Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards

arXiv 2506.20520 ↗First posted 25 Jun 2025cs.LG cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) is increasingly used to align large language models (LLMs). Off-policy methods offer greater implementation simplicity and data efficiency than on-policy techniques, but often result in suboptimal performance. In this work, we study the intermediate range of algorithms between off-policy RL and supervised fine-tuning by analyzing a simple off-policy REINFORCE algorithm, where the advantage is defined as A=r-V, with r a reward and V some tunable baseline. Intuitively, lowering V emphasizes high-reward samples, while raising it penalizes low-reward ones more heavily. We first provide a theoretical analysis of this off-policy REINFORCE algorithm, showing that when the baseline V lower-bounds the expected reward, the algorithm enjoys a policy improvement guarantee. Our analysis reveals that while on-policy updates can safely leverage both positive and negative signals, off-policy updates benefit from focusing more on positive rewards than on negative ones. We validate our findings experimentally in a controlled stochastic bandit setting and through fine-tuning state-of-the-art LLMs on reasoning tasks.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

24 of 37 references.

References · 24

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  12. 2014Convex Optimization: Algorithms and Complexitycited by 569
12 more
  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  2. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  3. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  4. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  5. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  6. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  7. 2020Text Generation by Learning from Demonstrationscited by 64
  8. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  9. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  10. 2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
  11. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 18
  12. 2025RL-finetuning LLMs from on- and off-policy data with a single algorithmcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 17.

29 papers.

Sort:

By area:

  1. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  2. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  3. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 49
  4. 2025ExGRPO: Learning to Reason from Experiencecited by 49
  5. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  6. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  7. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 13
  8. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
  9. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 11
  10. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  11. 2026Efficient RL Training for LLMs with Experience Replaycited by 8
  12. 2025Transforming Calabi-Yau Constructions: Generating New Calabi-Yau Manifolds with Transformerscited by 5
17 more
  1. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  2. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 4
  3. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 4
  4. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholdscited by 2
  5. 2026DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoningcited by 2
  6. 2026Beyond Variance: Prompt-Efficient RLVR via Rare-Event Amplification and Bidirectional Pairingcited by 2
  7. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 2
  8. 2026Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learningcited by 1
  9. 2026Distilling LLM Feedback for Lean Theorem Provingcited by 1
  10. 2026Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updatescited by 0
  11. 2026MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learningcited by 0
  12. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0
  13. 2026EAPO: Entropy-Driven Adaptive Positive-Negative Sample Weighting for Policy Optimization in Open-Ended QAcited by 0
  14. 2026What are Key Factors for Updates in RL for LLM Reasoning?cited by 0
  15. 2026Generating Special Triangulations with Transformerscited by 0
  16. 2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
  17. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policynesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  2. 2025Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
  3. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
  4. 2025Generalist Reward Models: Found Inside Large Language Models
  5. 2026Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think
  6. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization
  7. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning
  8. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices