# Policy Optimization with Stochastic Mirror Descent

arXiv: https://arxiv.org/abs/1906.10462 · HTML: https://mlsanity.com/paper/1906.10462/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2019.

## Built on by (most cited first, 18 of 18)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [A general sample complexity analysis of vanilla policy gradient](https://mlsanity.com/paper/2107.11433/) (2021)
- [Constrained Update Projection Approach to Safe Policy Optimization](https://mlsanity.com/paper/2209.07089/) (2022)
- [A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergence](https://mlsanity.com/paper/2301.13139/) (2023)
- [Bregman Gradient Policy Optimization](https://mlsanity.com/paper/2106.12112/) (2021)
- [Momentum-Based Policy Gradient with Second-Order Information](https://mlsanity.com/paper/2205.08253/) (2022)
- [On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spaces](https://mlsanity.com/paper/2201.12332/) (2022)
- [Learning to Constrain Policy Optimization with Virtual Trust Region](https://mlsanity.com/paper/2204.09315/) (2022)
- [FlagVNE: A Flexible and Generalizable Reinforcement Learning Framework for Network Resource Allocation](https://mlsanity.com/paper/2404.12633/) (2024)
- [Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Data](https://mlsanity.com/paper/2512.24056/) (2025)
- [Efficiently Escaping Saddle Points for Policy Optimization](https://mlsanity.com/paper/2311.08914/) (2023)
- [On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimization](https://mlsanity.com/paper/2401.12508/) (2024)
- [Mirror Descent Actor Critic via Bounded Advantage Learning](https://mlsanity.com/paper/2502.03854/) (2025)
- [Safe, Scalable, and Accurate Bayes Posterior Sampling for Large-Data Generalized Linear Mixed Models](https://mlsanity.com/paper/2604.26029/) (2026)
- [Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds](https://mlsanity.com/paper/2610.03165/) (2026)
- [Off-OAB: Off-Policy Policy Gradient Method with Optimal Action-Dependent Baseline](https://mlsanity.com/paper/2405.02572/) (2024)
- [Heavy-Ball Momentum Accelerated Actor-Critic With Function Approximation](https://mlsanity.com/paper/2408.06945/) (2024)
- [Mirror Descent on Riemannian Manifolds](https://mlsanity.com/paper/2603.17527/) (2026)

## Abstract (arXiv)

Improving sample efficiency has been a longstanding goal in reinforcement learning. This paper proposes mathttVRMPO algorithm: a sample efficient policy gradient method with stochastic mirror descent. In mathttVRMPO, a novel variance-reduced policy gradient estimator is presented to improve sample efficiency. We prove that the proposed mathttVRMPO needs only O(ε-3) sample trajectories to achieve an ε-approximate first-order stationary point, which matches the best sample complexity for policy optimization. The extensive experimental results demonstrate that mathttVRMPO outperforms the state-of-the-art policy gradient methods in various settings.