# Control Regularization for Reduced Variance Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.05380 · HTML: https://mlsanity.com/paper/1905.05380/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 May 2019.

## Built on by (most cited first, 30 of 37)

- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Contraction Theory for Nonlinear Stability Analysis and Learning-based Control: A Tutorial Overview](https://mlsanity.com/paper/2110.00675/) (2021)
- [LyaNet: A Lyapunov Framework for Training Neural ODEs](https://mlsanity.com/paper/2202.02526/) (2022)
- [The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models](https://mlsanity.com/paper/2108.06084/) (2021)
- [Programmatically Grounded, Compositionally Generalizable Robotic Manipulation](https://mlsanity.com/paper/2304.13826/) (2023)
- [A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios](https://mlsanity.com/paper/2412.03920/) (2024)
- [Robot Learning of Mobile Manipulation with Reachability Behavior Priors](https://mlsanity.com/paper/2203.04051/) (2022)
- [Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics](https://mlsanity.com/paper/2107.09822/) (2021)
- [Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own](https://mlsanity.com/paper/2310.02635/) (2023)
- [Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings](https://mlsanity.com/paper/2305.16614/) (2023)
- [Beyond Black-Box Advice: Learning-Augmented Algorithms for MDPs with Q-Value Predictions](https://mlsanity.com/paper/2307.10524/) (2023)
- [Reinforcement Learning Your Way: Agent Characterization through Policy Regularization](https://mlsanity.com/paper/2201.10003/) (2022)
- [Equipping Black-Box Policies with Model-Based Advice for Stable Nonlinear Control](https://mlsanity.com/paper/2206.01341/) (2022)
- [Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems](https://mlsanity.com/paper/2404.15199/) (2024)
- [Contextualized Hybrid Ensemble Q-learning: Learning Fast with Control Priors](https://mlsanity.com/paper/2406.19768/) (2024)
- [Reinforcement Learning with Prior Policy Guidance for Motion Planning of Dual-Arm Free-Floating Space Robot](https://mlsanity.com/paper/2209.01434/) (2022)
- [A3C-S: Automated Agent Accelerator Co-Search towards Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2106.06577/) (2021)
- [Certifiably Robust Reinforcement Learning through Model-Based Abstract Interpretation](https://mlsanity.com/paper/2301.11374/) (2023)
- [Robust Reinforcement Learning under model misspecification](https://mlsanity.com/paper/2103.15370/) (2021)
- [Simplex-enabled Safe Continual Learning Machine](https://mlsanity.com/paper/2409.05898/) (2024)
- [Stochastic Intervention for Causal Inference via Reinforcement Learning](https://mlsanity.com/paper/2105.13514/) (2021)
- [Coordinate-wise Control Variates for Deep Policy Gradients](https://mlsanity.com/paper/2107.04987/) (2021)
- [Barrier Certified Safety Learning Control: When Sum-of-Square Programming Meets Reinforcement Learning](https://mlsanity.com/paper/2206.07915/) (2022)
- [Reinforcement Learning-Based Air Traffic Deconfliction](https://mlsanity.com/paper/2301.01861/) (2023)
- [Learning-Augmented Decentralized Online Convex Optimization in Networks](https://mlsanity.com/paper/2306.10158/) (2023)
- [Stable Modular Control via Contraction Theory for Reinforcement Learning](https://mlsanity.com/paper/2311.03669/) (2023)
- [Out-of-Distribution-Aware Electric Vehicle Charging](https://mlsanity.com/paper/2311.05941/) (2023)
- [OGMP: Oracle Guided Multi-mode Policies for Agile and Versatile Robot Control](https://mlsanity.com/paper/2403.04205/) (2024)
- [Shape Arithmetic Expressions: Advancing Scientific Discovery Beyond Closed-Form Equations](https://mlsanity.com/paper/2404.09788/) (2024)
- [CHEQ-ing the Box: Safe Variable Impedance Learning for Robotic Polishing](https://mlsanity.com/paper/2501.07985/) (2025)

## Abstract (arXiv)

Dealing with high variance is a significant challenge in model-free reinforcement learning (RL). Existing methods are unreliable, exhibiting high variance in performance from run to run using different initializations/seeds. Focusing on problems arising in continuous control, we propose a functional regularization approach to augmenting model-free RL. In particular, we regularize the behavior of the deep policy to be similar to a policy prior, i.e., we regularize in function space. We show that functional regularization yields a bias-variance trade-off, and propose an adaptive tuning strategy to optimize this trade-off. When the policy prior has control-theoretic stability guarantees, we further show that this regularization approximately preserves those stability guarantees throughout learning. We validate our approach empirically on a range of settings, and demonstrate significantly reduced variance, guaranteed dynamic stability, and more efficient learning than deep RL alone.