mlsanityData to 14 Oct 2026

Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs

arXiv 2503.14286 ↗First posted 18 Mar 2025cs.LGcited by 61

Not on the map. Why?

Cited by 37 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a new algorithm for fine-tuning large language models using reinforcement learning. Tapered Off-Policy REINFORCE (TOPR) uses an asymmetric, tapered variant of importance sampling to speed up learning while maintaining stable learning dynamics, even without the use of KL regularization. TOPR can be applied in a fully offline fashion, allows the handling of positive and negative examples in a unified framework, and benefits from the implementational simplicity that is typical of Monte Carlo algorithms. We demonstrate the effectiveness of our approach with a series of experiments on the GSM8K and MATH reasoning benchmarks, finding performance gains for training both a model for solution generation and as a generative verifier. We show that properly leveraging positive and negative examples alike in the off-policy regime simultaneously increases test-time accuracy and training data efficiency, all the while avoiding the ``wasted inference'' that comes with discarding negative examples. We find that this advantage persists over multiple iterations of training and can be amplified by dataset curation techniques, enabling us to match 70B-parameter model performance with 8B language models. As a corollary to this work, we find that REINFORCE's baseline parameter plays an important and unexpected role in defining dataset composition in the presence of negative examples, and is consequently critical in driving off-policy performance.

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

45 of 59 references.

References · 45

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
33 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  3. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  9. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  10. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  11. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  12. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  13. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  14. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  15. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  16. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  17. 2016Variance-based regularization with convex objectivescited by 155
  18. 2024Generative Reward Modelscited by 148
  19. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  20. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  21. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  22. 2020Text Generation by Learning from Demonstrationscited by 64
  23. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  24. 2024Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrievalcited by 26
  25. 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 18
  26. 2020An operator view of policy gradient methodscited by 17
  27. 2024Corpus-Steered Query Expansion with Large Language Modelscited by 17
  28. 2022Simulating Bandit Learning from User Feedback for Extractive Question Answeringcited by 14
  29. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  30. 2024Generative Query Reformulation Using Ensemble Prompting, Document Fusion, and Relevance Feedbackcited by 5
  31. 2024Averaging log-likelihoods in direct alignmentcited by 5
  32. 2016Tighter bounds lead to improved classifierscited by 3
  33. 2016Efficient iterative policy optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 34; 2026: 27.

61 papers.

Sort:

By area:

  1. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  4. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  5. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  6. 2025Reinforcement Learning from Human Feedbackcited by 58
  7. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
  8. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  9. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  10. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 32
  11. 2025Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystemcited by 30
  12. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
38 more
  1. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 23
  2. 2025How to Train Your LLM Web Agent: A Statistical Diagnosiscited by 19
  3. 2025Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Modelscited by 18
  4. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 15
  5. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  6. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweightingcited by 12
  7. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 11
  8. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  9. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 9
  10. 2026Rethinking Muon Beyond Pretraining: Spectral Failures and High-Pass Remedies for VLA and RLVRcited by 8
  11. 2025It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RLcited by 7
  12. 2025Polychromic Objectives for Reinforcement Learningcited by 7
  13. 2025The Era of Agentic Organization: Learning to Organize with Language Modelscited by 7
  14. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimizationcited by 7
  15. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learningcited by 7
  16. 2025Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoningcited by 6
  17. 2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functionscited by 6
  18. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuningcited by 6
  19. 2025Enhancing Large Language Models through Structured Reasoningcited by 5
  20. 2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoningcited by 5
  21. 2026Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correctioncited by 5
  22. 2025GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learningcited by 4
  23. 2025Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scalingcited by 4
  24. 2025Fast LLM Post-training via Decoupled and Fastest-of-N Speculationcited by 4
  25. 2026Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewardscited by 4
  26. 2025Learning to Reason Efficiently with Discounted Reinforcement Learningcited by 3
  27. 2025CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Samplingcited by 3
  28. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 3
  29. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learningcited by 3
  30. 2026Score Centering Stabilizes Off-policy Reinforcement Learningcited by 3
  31. 2025Understanding Reinforcement Learning for Model Training, and future directions with GRAPEcited by 2
  32. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  33. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  34. 2026REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learningcited by 2
  35. 2025GIFT: Group-Relative Implicit Fine-Tuning Integrates GRPO with DPO and UNAcited by 1
  36. 2026It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branchescited by 1
  37. 2026Learning to Solve Hard Problems in RL for LLMs by Never Giving Upcited by 1
  38. 2025Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rolloutscited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  2. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
  3. 2026Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignment
  4. 2025RL-finetuning LLMs from on- and off-policy data with a single algorithm
  5. 2025RAPID: An Efficient Reinforcement Learning Algorithm for Small Language Models
  6. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
  7. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning
  8. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning