mlsanityData to 7 Oct 2026

Off-Policy Value-Based Reinforcement Learning for Large Language Models

arXiv 2603.23355 ↗First posted 24 Mar 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Improving data utilization efficiency is critical for scaling reinforcement learning (RL) for long-horizon tasks where generating trajectories is expensive. However, the dominant RL methods for LLMs are largely on-policy: they update each batch of data only once, discard it, and then collect fresh samples, resulting in poor sample efficiency. In this work, we explore an alternative value-based RL framework for LLMs that naturally enables off-policy learning. We propose ReVal, a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification. ReVal naturally supports replay-buffer-based training, allowing efficient reuse of past trajectories. Experiments on standard mathematical reasoning benchmarks show that ReVal not only converges faster but also outperforms GRPO in final performance. On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO. These results suggest that value-based RL is a practical alternative to policy-based methods for LLM training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 42 references that are in the map. First posted on arXiv 24 Mar 2026.

Builds on

30 of 42 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026LLMs Can Learn to Reason Via Off-Policy RL22 Feb
  2. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?1 Oct
  3. 2025Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards29 Sep
  4. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training25 Sep
  5. 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning10 Jul
  6. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  7. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  8. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning21 May
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 18

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2015Prioritized Experience Replaycited by 1,070
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  10. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  11. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  12. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
6 more
  1. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  2. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  3. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  4. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  5. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  6. 2026Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes Itcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model
  2. 2026REVES: REvision and VErification--Augmented Training for Test-Time Scaling
  3. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
  4. 2026REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learning
  5. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  6. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  7. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  8. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards