# Reward-Weighted Regression Converges to a Global Optimum

arXiv: https://arxiv.org/abs/2107.09088 · HTML: https://mlsanity.com/paper/2107.09088/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 44 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 5 of 5)

- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Relative Entropy Regularized Policy Iteration](https://mlsanity.com/paper/1812.02256/) (2018)
- [Hierarchical Policy Search via Return-Weighted Density Estimation](https://mlsanity.com/paper/1711.10173/) (2017)

## Built on by (most cited first, 6 of 6)

- [Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resets](https://mlsanity.com/paper/2205.06595/) (2022)
- [UniFL: Improve Latent Diffusion Model via Unified Feedback Learning](https://mlsanity.com/paper/2404.05595/) (2024)
- [A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectories](https://mlsanity.com/paper/2311.01329/) (2023)
- [Generative Bayesian Optimization: Generative Models as Acquisition Functions](https://mlsanity.com/paper/2510.25240/) (2025)
- [Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness](https://mlsanity.com/paper/2609.36945/) (2026)
- [On the Convergence of Success Conditioning for Policy Optimization](https://mlsanity.com/paper/2610.03642/) (2026)

## Abstract (arXiv)

Reward-Weighted Regression (RWR) belongs to a family of widely known iterative Reinforcement Learning algorithms based on the Expectation-Maximization framework. In this family, learning at each iteration consists of sampling a batch of trajectories using the current policy and fitting a new policy to maximize a return-weighted log-likelihood of actions. Although RWR is known to yield monotonic improvement of the policy under certain circumstances, whether and under which conditions RWR converges to the optimal policy have remained open questions. In this paper, we provide for the first time a proof that RWR converges to a global optimum when no function approximation is used, in a general compact setting. Furthermore, for the simpler case with finite state and action spaces we prove R-linear convergence of the state-value function to the optimum.