# Adversarially Regularized Policy Learning Guided by Trajectory Optimization

arXiv: https://arxiv.org/abs/2109.07627 · HTML: https://mlsanity.com/paper/2109.07627/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 54 references that are in the map.
- First posted on arXiv 16 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization](https://mlsanity.com/paper/1911.03437/) (2019)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [Generalized Inner Loop Meta-Learning](https://mlsanity.com/paper/1910.01727/) (2019)
- [Lipschitz Continuity in Model-based Reinforcement Learning](https://mlsanity.com/paper/1804.07193/) (2018)
- [Task-Relevant Adversarial Imitation Learning](https://mlsanity.com/paper/1910.01077/) (2019)
- [Crocoddyl: An Efficient and Versatile Framework for Multi-Contact Optimal Control](https://mlsanity.com/paper/1909.04947/) (2019)
- [Adversarial Training is Not Ready for Robot Learning](https://mlsanity.com/paper/2103.08187/) (2021)
- [A Closer Look at Accuracy vs. Robustness](https://mlsanity.com/paper/2003.02460/) (2020)
- [Adversarial Feature Training for Generalizable Robotic Visuomotor Control](https://mlsanity.com/paper/1909.07745/) (2019)
- [Robust Trajectory Optimization over Uncertain Terrain with Stochastic Complementarity](https://mlsanity.com/paper/2009.12409/) (2020)
- [On Value Discrepancy of Imitation Learning](https://mlsanity.com/paper/1911.07027/) (2019)
- [SyDeBO: Symbolic-Decision-Embedded Bilevel Optimization for Long-Horizon Manipulation in Dynamic Environments](https://mlsanity.com/paper/2010.11078/) (2020)
- [Accelerated ADMM based Trajectory Optimization for Legged Locomotion with Coupled Rigid Body Dynamics](https://mlsanity.com/paper/2003.08019/) (2020)
- [Online Trajectory Planning Through Combined Trajectory Optimization and Function Approximation: Application to the Exoskeleton Atalante](https://mlsanity.com/paper/1910.00514/) (2019)
- [ARCH: Efficient Adversarial Regularized Training with Caching](https://mlsanity.com/paper/2109.07048/) (2021)

## Built on by (most cited first, 4 of 4)

- [Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms](https://mlsanity.com/paper/2310.10810/) (2023)
- [A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approaches](https://mlsanity.com/paper/2404.02817/) (2024)
- [Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint](https://mlsanity.com/paper/2206.04569/) (2022)
- [Robust Behavior Cloning Via Global Lipschitz Regularization](https://mlsanity.com/paper/2506.19250/) (2025)

## Abstract (arXiv)

Recent advancement in combining trajectory optimization with function approximation (especially neural networks) shows promise in learning complex control policies for diverse tasks in robot systems. Despite their great flexibility, the large neural networks for parameterizing control policies impose significant challenges. The learned neural control policies are often overcomplex and non-smooth, which can easily cause unexpected or diverging robot motions. Therefore, they often yield poor generalization performance in practice. To address this issue, we propose adVErsarially Regularized pOlicy learNIng guided by trajeCtory optimizAtion (VERONICA) for learning smooth control policies. Specifically, our proposed approach controls the smoothness (local Lipschitz continuity) of the neural control policies by stabilizing the output control with respect to the worst-case perturbation to the input state. Our experiments on robot manipulation show that our proposed approach not only improves the sample efficiency of neural policy learning but also enhances the robustness of the policy against various types of disturbances, including sensor noise, environmental uncertainty, and model mismatch.