# On the Convergence of Success Conditioning for Policy Optimization

arXiv: https://arxiv.org/abs/2610.03642 · HTML: https://mlsanity.com/paper/2610.03642/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs](https://mlsanity.com/paper/1909.02769/) (2019)
- [When does return-conditioned supervised learning work for offline reinforcement learning?](https://mlsanity.com/paper/2206.01079/) (2022)
- [Self-Improving Embodied Foundation Models](https://mlsanity.com/paper/2509.15155/) (2025)
- [Imitating Past Successes can be Very Suboptimal](https://mlsanity.com/paper/2206.03378/) (2022)
- [On the Role of Temperature Sampling in Test-Time Scaling](https://mlsanity.com/paper/2510.02611/) (2025)
- [Reward-Weighted Regression Converges to a Global Optimum](https://mlsanity.com/paper/2107.09088/) (2021)
- [Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success](https://mlsanity.com/paper/2601.18175/) (2026)

## Abstract (arXiv)

Success conditioning is a strategy for improving decision-making policies in stochastic environments; it updates a policy by increasing the probability of taking actions that yield successful outcomes. Success conditioning is common to many reinforcement learning applications, yet its limiting behavior and convergence rates are not well understood. In this work, we demonstrate that success conditioning converges to an optimal policy on a broad class of Markov decision processes (MDPs). We also derive convergence rates in some common settings. For discounted MDPs, we prove convergence within O(1/varepsilonp) iterations to an varepsilon-optimal policy, where the exponent p depends on problem data. For single-period MDPs, such a policy is obtained within O(log(1/varepsilon)) iterations.