mlsanityData to 14 Oct 2026

TPPO: Tracking-Value PPO For Language Model Reasoning

arXiv 2610.16794 ↗First posted 12 Oct 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Proximal Policy Optimization (PPO) is widely used in reinforcement learning with verifiable rewards for language-model reasoning. While value evolution along trajectory characterizes relative token-level credit, yet standard critics often fail to track it under sparse terminal rewards. To model this evolution, we analyze the relation between successive prefix values and show that they admit a Bayesian decomposition with additive updates in log-odds space. We use this additive structure to construct Tracking-Value PPO (method), a cumulative value parameterization that sums position-wise scalar outputs into value logits. These outputs are learned through terminal-outcome supervision on the reconstructed values. method improves aggregate mathematical reasoning scores over PPO by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively, while improving out-of-distribution generalization. Gains extend to Qwen3-4B, Qwen2.5-Math-1.5B, Qwen3.5-9B and to countdown, and scientific coding tasks. Further analysis shows more accurate tracking of value evolution and more reliable advantage estimates. Our code is publicly available in https://github.com/LINs-lab/TPPO.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 40 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

33 of 40 references.

Closest ancestors · 12

Same direction, newest first

  1. 2026SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks10 Apr
  2. 2025TreePO: Bridging the Gap of Policy Optimization and Efficacy and Inference Efficiency with Heuristic Tree-based Modeling24 Aug
  3. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning10 Mar
  6. 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
  7. 2025Process Reinforcement through Implicit Rewards3 Feb
  8. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  9. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning10 Oct
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  11. 2023Let's Verify Step by Step31 May
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2021Measuring Coding Challenge Competence With APPScited by 902
9 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  3. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 239
  4. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  5. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  6. 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 178
  7. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  8. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 80
  9. 2026Start Classifying: Categorical Critics for LLM Reinforcement Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning
  2. 2026DISPO: Enhancing Training Efficiency and Stability in Reinforcement Learning for Large Language Model Mathematical Reasoning
  3. 2026Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
  4. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
  5. 2025Truncated Proximal Policy Optimization
  6. 2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States
  7. 2026VIMPO: Value-Implicit Policy Optimization for LLMs
  8. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training