Target Policy Optimization
Not on the map. Why?
Abstract (arXiv)
In RL, given a prompt, we sample a group of completions from a model and score them. Two questions follow: which completions should gain probability mass, and how should the parameters move to realize that change? Standard policy-gradient methods answer both at once, so the update can overshoot or undershoot depending on the learning rate, clipping, and other optimizer choices. We introduce Target Policy Optimization (TPO), which separates the two questions. Given scored completions, TPO constructs a target distribution qi propto pi,old exp(ui) and fits the policy to it by cross-entropy. The loss gradient on sampled-completion logits is pθ - q, which vanishes once the policy matches the target. On tabular bandits, transformer sequence tasks, and billion-parameter LLM RLVR, TPO matches PG, PPO, GRPO, and DG on easy tasks and substantially outperforms them under sparse reward. Code is available at https://github.com/JeanKaddour/tpo.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 7 Apr 2026.
Builds on
25 of 38 references.
References · 25
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2015Trust Region Policy Optimizationcited by 2,503
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
- 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
- 2018Maximum a Posteriori Policy Optimisationcited by 318
13 more
- 2019A Theory of Regularized Markov Decision Processescited by 279
- 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
- 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
- 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
- 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
- 2020Mirror Descent Policy Optimizationcited by 102
- 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
- 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
- 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
- 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 50
- 2026Maximum Likelihood Reinforcement Learningcited by 29
- 2024Learning from negative feedback, or positive feedback or bothcited by 13
- 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 12
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
- 2025Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards
- 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
- 2026GOPO: Policy Optimization using Ranked Rewards
- 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity
- 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution
- 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
- 2026Post-Training with Policy Gradients: Optimality and the Base Model Barrier