# Policy Gradient for LQR with Domain Randomization

arXiv: https://arxiv.org/abs/2503.24371 · HTML: https://mlsanity.com/paper/2503.24371/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 31 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 19 of 19)

- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator](https://mlsanity.com/paper/1801.05039/) (2018)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [Naive Exploration is Optimal for Online LQR](https://mlsanity.com/paper/2001.09576/) (2020)
- [Understanding Domain Randomization for Sim-to-real Transfer](https://mlsanity.com/paper/2110.03239/) (2021)
- [Deep Drone Racing: From Simulation to Reality with Domain Randomization](https://mlsanity.com/paper/1905.09727/) (2019)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergence](https://mlsanity.com/paper/1910.09496/) (2019)
- [Stabilizing Dynamical Systems via Policy Gradient Methods](https://mlsanity.com/paper/2110.06418/) (2021)
- [Model-free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approach](https://mlsanity.com/paper/2308.11743/) (2023)
- [Globally Convergent Policy Search over Dynamic Filters for Output Estimation](https://mlsanity.com/paper/2202.11659/) (2022)
- [Learning robust control for LQR systems with multiplicative noise via policy gradient](https://mlsanity.com/paper/1905.13547/) (2019)
- [PAC Reinforcement Learning without Real-World Feedback](https://mlsanity.com/paper/1909.10449/) (2019)
- [Blue River Controls: A toolkit for Reinforcement Learning Control Systems on Hardware](https://mlsanity.com/paper/2001.02254/) (2020)
- [Domain Randomization is Sample Efficient for Linear Quadratic Control](https://mlsanity.com/paper/2502.12310/) (2025)
- [Global Convergence Using Policy Gradient Methods for Model-free Markovian Jump Linear Quadratic Control](https://mlsanity.com/paper/2111.15228/) (2021)
- [Sparse optimal control of networks with multiplicative noise via policy gradient](https://mlsanity.com/paper/1905.13548/) (2019)

## Built on by (most cited first, 5 of 5)

- [Policy Gradient Bounds in Multitask LQR](https://mlsanity.com/paper/2509.19266/) (2025)
- [Gradient Dominance in the Linear Quadratic Regulator: A Unified Analysis for Continuous-Time and Discrete-Time Systems](https://mlsanity.com/paper/2602.22577/) (2026)
- [Multitask LQG Control: Performance and Generalization Bounds](https://mlsanity.com/paper/2604.16730/) (2026)
- [On Globally Optimal Stochastic Policy Gradient Methods for Domain Randomized LQR Synthesis](https://mlsanity.com/paper/2603.14197/) (2026)
- [Policy Gradient over History-Dependent Policy Classes for LQR with Domain Randomization](https://mlsanity.com/paper/2609.16300/) (2026)

## Abstract (arXiv)

Domain randomization (DR) enables sim-to-real transfer by training controllers on a distribution of simulated environments, with the goal of achieving robust performance in the real world. Although DR is widely used in practice and is often solved using simple policy gradient (PG) methods, understanding of its theoretical guarantees remains limited. Toward addressing this gap, we provide the first convergence analysis of PG methods for domain-randomized linear quadratic regulation (LQR). We show that PG converges globally to the minimizer of a finite-sample approximation of the DR objective under suitable bounds on the heterogeneity of the sampled systems. We also quantify the sample-complexity associated with achieving a small performance gap between the sample-average and population-level objectives. Additionally, we propose and analyze a discount-factor annealing algorithm that obviates the need for an initial jointly stabilizing controller, which may be challenging to find. Empirical results support our theoretical findings and highlight promising directions for future work, including risk-sensitive DR formulations and stochastic PG algorithms.