mlsanityData to 7 Oct 2026

Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning

arXiv 2602.20722 ↗First posted 24 Feb 2026cs.AIcited by 8

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-training. In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training. It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement. Extensive experiments further demonstrate that BAPO achieves an average 12.5% improvement over GRPO across mathematics, planning, and visual reasoning tasks. Crucially, BAPO successfully resolves 40.7% of problems that base models consistently fail to solve.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 41 references that are in the map. First posted on arXiv 24 Feb 2026.

Builds on

33 of 41 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control5 Aug
  2. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model9 Jul
  3. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
  4. 2025RePO: Replay-Enhanced Policy Optimization11 Jun
  5. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  6. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay5 Jun
  7. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  9. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training28 May
  10. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
  11. 2025Self-Evolving Curriculum for LLM Reasoning20 May
  12. 2025Learning to Reason under Off-Policy Guidance21 Apr
6 more
  1. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov

Other references · 15

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  8. 2025Group Sequence Policy Optimizationcited by 788
  9. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  10. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  11. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 381
  12. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
3 more
  1. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
  2. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  3. 2025GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learningcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Modelscited by 7
  2. 2026ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compressioncited by 2
  3. 2026Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradientscited by 1
  4. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 1
  5. 2026Headroom-Drift Replay: A Primitive for Principled Replay Control in GRPOcited by 1
  6. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0
  7. 2026Turning Off-Policy Tokens On-Policy: A Plug-in Approach for Improving LLM Alignmentcited by 0
  8. 2026Mitigating the Length-Scaling Tax with Online Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
  2. 2024Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization
  3. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models
  4. 2026Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
  5. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  6. 2026Exploration-Driven Optimization for Test-Time Large Language Model Reasoning
  7. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  8. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping