mlsanityData to 14 Oct 2026

Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success

arXiv 2601.18175 ↗First posted 26 Jan 2026cs.AI cs.LG cs.SYcited by 4

Not on the map. Why?

Abstract (arXiv)

A widely used technique for improving policies is success conditioning, in which one collects trajectories, identifies those that achieve a desired outcome, and updates the policy to imitate the actions taken along successful trajectories. This principle appears under many names -- rejection sampling with SFT, goal-conditioned RL, Decision Transformers -- yet what optimization problem it solves, if any, has remained unclear. We prove that success conditioning exactly solves a trust-region optimization problem, maximizing policy improvement subject to a chi2 divergence constraint whose radius is determined automatically by the data. This yields an identity: relative policy improvement, the magnitude of policy change, and a quantity we call action-influence -- measuring how random variation in action choices affects success rates -- are exactly equal at every state. Success conditioning thus emerges as a conservative improvement operator. Exact success conditioning cannot degrade performance or induce dangerous distribution shift, but when it fails, it does so observably, by hardly changing the policy at all. We apply our theory to the common practice of return thresholding, showing this can amplify improvement, but at the cost of potential misalignment with the true objective.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 26 Jan 2026.

Builds on

28 of 39 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
16 more
  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  2. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  3. 2018Maximum a Posteriori Policy Optimisationcited by 328
  4. 2019A Theory of Regularized Markov Decision Processescited by 286
  5. 2020Critic Regularized Regressioncited by 275
  6. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  7. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
  8. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  9. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  10. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  11. 2020Mirror Descent Policy Optimizationcited by 106
  12. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  13. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 63
  14. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  15. 2025Self-Improving Embodied Foundation Modelscited by 32
  16. 2022Imitating Past Successes can be Very Suboptimalcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026THINKSAFE: Self-Generated Safety Alignment for Reasoning Modelscited by 10
  2. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policynesscited by 0
  3. 2026On the Convergence of Success Conditioning for Policy Optimizationcited by 0
  4. 2026Co-Evolving Acting and Forecasting for Language Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias
  2. 2026Towards Optimal Policy Improvement
  3. 2026Delightful Policy Gradient
  4. 2026On the "Causality" Step in Policy Gradient Derivations: A Pedagogical Reconciliation of Full Return and Reward-to-Go
  5. 2026Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling
  6. 2026Policy Improvement Reinforcement Learning
  7. 2026Beyond Importance Sampling: Rejection-Gated Policy Optimization
  8. 2026On Training in Imagination