mlsanityData to 14 Oct 2026

Bellman Policy Optimization

arXiv 2609.15987 ↗First posted 14 Sep 2026cs.LG cs.CL math.OCcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 36 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

32 of 36 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026VIMPO: Value-Implicit Policy Optimization for LLMs18 Jun
  2. 2026Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning9 Jun
  3. 2026Rethinking the Divergence Regularization in LLM RL8 Jun
  4. 2026FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization20 Mar
  5. 2026Rethinking the Trust Region in LLM Reinforcement Learning4 Feb
  6. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model21 Oct
  7. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  8. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  9. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning23 May
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025Group Sequence Policy Optimizationcited by 835
  4. 2019A Theory of Regularized Markov Decision Processescited by 286
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  6. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  7. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  8. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  9. 2025Phi-4-reasoning Technical Reportcited by 151
  10. 2024Token-level Direct Preference Optimizationcited by 135
  11. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  12. 2020Mirror Descent Policy Optimizationcited by 106
7 more
  1. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  2. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 72
  3. 2025Defeating the Training-Inference Mismatch via FP16cited by 66
  4. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 65
  5. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  6. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 41
  7. 2026Tmax: A simple recipe for terminal agentscited by 33

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
  2. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  3. 2026Re2: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving
  4. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  5. 2026One-Way Policy Optimization for Self-Evolving LLMs
  6. 2026Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
  7. 2024Teaching Large Language Models to Reason with Reinforcement Learning
  8. 2025Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies