mlsanityData to 14 Oct 2026

Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning

arXiv 2505.15311 ↗First posted 21 May 2025cs.LG cs.AI cs.CLcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Policy-based methods currently dominate reinforcement learning (RL) pipelines for large language model (LLM) reasoning, leaving value-based approaches largely unexplored. We revisit the classical paradigm of Bellman Residual Minimization and introduce Trajectory Bellman Residual Minimization (TBRM), an algorithm that naturally adapts this idea to LLMs, yielding a simple yet effective off-policy algorithm that optimizes a single trajectory-level Bellman objective using the model's own logits as Q-values. TBRM removes the need for critics, importance-sampling ratios, or clipping, and operates with only one rollout per prompt. We prove convergence to the near-optimal KL-regularized policy from arbitrary off-policy data via an improved change-of-trajectory-measure analysis. Experiments on standard mathematical-reasoning benchmarks show that TBRM consistently outperforms policy-based baselines, like PPO and GRPO, with comparable or lower computational and memory overhead. Our results indicate that value-based RL might be a principled and efficient alternative for enhancing reasoning capabilities in LLMs.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 71 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

34 of 71 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  2. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  9. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  10. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective14 Feb
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
5 more
  1. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  2. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoning20 Dec
  3. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization11 Oct
  4. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  9. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  10. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  11. 2017A unified view of entropy-regularized Markov decision processescited by 182
  12. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
5 more
  1. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  2. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  3. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  4. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  5. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 8.

11 papers.

  1. 2025NFT: Bridging Supervised Learning and Reinforcement Learning in Math Reasoningcited by 38
  2. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 5
  3. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholdscited by 4
  4. 2025Is RL fine-tuning harder than regression? A PDE learning approach for diffusion modelscited by 3
  5. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  6. 2026Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov Statescited by 1
  7. 2026Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distributioncited by 0
  8. 2026Off-Policy Value-Based Reinforcement Learning for Large Language Modelscited by 0
  9. 2026When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?cited by 0
  10. 2026Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learningcited by 0
  11. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  2. 2026R2PO: Decoupling Rollout and Inference Policies for LLM Reasoning
  3. 2026Bellman Policy Optimization
  4. 2026Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning
  5. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  6. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  7. 2025ShiQ: Bringing back Bellman to LLMs
  8. 2024Efficient Reinforcement Learning with Large Language Model Priors