mlsanityData to 7 Oct 2026

On Robust Reinforcement Learning with Lipschitz-Bounded Policy Networks

arXiv 2405.11432 ↗First posted 19 May 2024cs.LG cs.SY eess.SYcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper presents a study of robust policy networks in deep reinforcement learning. We investigate the benefits of policy parameterizations that naturally satisfy constraints on their Lipschitz bound, analyzing their empirical performance and robustness on two representative problems: pendulum swing-up and Atari Pong. We illustrate that policy networks with smaller Lipschitz bounds are more robust to disturbances, random noise, and targeted adversarial attacks than unconstrained policies composed of vanilla multi-layer perceptrons or convolutional neural networks. However, the structure of the Lipschitz layer is important. We find that the widely-used method of spectral normalization is too conservative and severely impacts clean performance, whereas more expressive Lipschitz layers such as the recently-proposed Sandwich layer can achieve improved robustness without sacrificing clean performance.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 19 May 2024.

Builds on

25 of 34 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  6. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  7. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  8. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  9. 2017Adversarial Attacks on Neural Network Policiescited by 314
  10. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  11. 2018Adversarial Attacks and Defences: A Surveycited by 259
  12. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
13 more
  1. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  2. 2020Training robust neural networks using Lipschitz boundscited by 98
  3. 2020Robust Deep Reinforcement Learning through Adversarial Losscited by 73
  4. 2019advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorchcited by 69
  5. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 58
  6. 2023A Unified Algebraic Perspective on Lipschitz Neural Networkscited by 58
  7. 2020Lipschitz Bounded Equilibrium Networkscited by 49
  8. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 49
  9. 2021CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothingcited by 41
  10. 2021Policy Smoothing for Provably Robust Reinforcement Learningcited by 39
  11. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  12. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  13. 2023Improve Robustness of Reinforcement Learning against Observation Perturbations via l_∞ Lipschitz Policy Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Improved Scalable Lipschitz Bounds for Deep Neural Networkscited by 2
  2. 2025R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networkscited by 2
  3. 2025A Scalable Approach for Safe and Robust Learning via Lipschitz-Constrained Networkscited by 0
  4. 2026Scalable Incremental Robustness Analysis of Neural Network Feedback Systemscited by 0