# L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning

arXiv: https://arxiv.org/abs/2202.07152 · HTML: https://mlsanity.com/paper/2202.07152/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Deep Exploration via Bootstrapped DQN](https://mlsanity.com/paper/1602.04621/) (2016)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations](https://mlsanity.com/paper/2003.08938/) (2020)
- [Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones](https://mlsanity.com/paper/2010.15920/) (2020)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [Robust Reinforcement Learning on State Observations with Learned Optimal Adversary](https://mlsanity.com/paper/2101.08452/) (2021)
- [Regularizing Action Policies for Smooth Control with Reinforcement Learning](https://mlsanity.com/paper/2012.06644/) (2020)
- [Connecting Generative Adversarial Networks and Actor-Critic Methods](https://mlsanity.com/paper/1610.01945/) (2016)
- [Squareplus: A Softplus-Like Algebraic Rectifier](https://mlsanity.com/paper/2112.11687/) (2021)
- [t-Soft Update of Target Network for Deep Reinforcement Learning](https://mlsanity.com/paper/2008.10861/) (2020)
- [Proximal Policy Optimization with Relative Pearson Divergence](https://mlsanity.com/paper/2010.03290/) (2020)
- [How to Train your Quadrotor: A Framework for Consistently Smooth and Responsive Flight Control via Reinforcement Learning](https://mlsanity.com/paper/2012.06656/) (2020)
- [Temporal Regularization in Markov Decision Process](https://mlsanity.com/paper/1811.00429/) (2018)
- [Coarse-Grained Smoothness for RL in Metric Spaces](https://mlsanity.com/paper/2110.12276/) (2021)

## Built on by (most cited first, 27 of 27)

- [Learning Humanoid Standing-up Control across Diverse Postures](https://mlsanity.com/paper/2502.08378/) (2025)
- [Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies](https://mlsanity.com/paper/2410.11825/) (2024)
- [PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System](https://mlsanity.com/paper/2510.11072/) (2025)
- [Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning](https://mlsanity.com/paper/2407.04315/) (2024)
- [Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies](https://mlsanity.com/paper/2410.16632/) (2024)
- [Consolidated Adaptive T-soft Update for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.12504/) (2022)
- [Reward Bonuses with Gain Scheduling Inspired by Iterative Deepening Search](https://mlsanity.com/paper/2212.10765/) (2022)
- [AGILE: A Comprehensive Workflow for Humanoid Loco-Manipulation Learning](https://mlsanity.com/paper/2603.20147/) (2026)
- [Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint](https://mlsanity.com/paper/2303.04356/) (2023)
- [Revisiting Experience Replayable Conditions](https://mlsanity.com/paper/2402.10374/) (2024)
- [LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World](https://mlsanity.com/paper/2409.19617/) (2024)
- [Mastering Diverse, Unknown, and Cluttered Tracks for Robust Vision-Based Drone Racing](https://mlsanity.com/paper/2512.09571/) (2025)
- [Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States](https://mlsanity.com/paper/2601.18479/) (2026)
- [Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods](https://mlsanity.com/paper/2601.22970/) (2026)
- [HG2P: Hippocampus-inspired High-reward Graph and Model-Free Q-Gradient Penalty for Path Planning and Motion Control](https://mlsanity.com/paper/2410.09505/) (2024)
- [Weber-Fechner Law in Temporal Difference learning derived from Control as Inference](https://mlsanity.com/paper/2412.21004/) (2024)
- [Implicit Constraint-Aware Off-Policy Correction for Offline Reinforcement Learning](https://mlsanity.com/paper/2506.14058/) (2025)
- [Difference-Aware Retrieval Policies for Imitation Learning](https://mlsanity.com/paper/2606.09758/) (2026)
- [Lipschitz-Regularized Critics Lead to Policy Robustness Against Transition Dynamics Uncertainty](https://mlsanity.com/paper/2404.13879/) (2024)
- [Deep Reinforcement Learning for Day-to-day Dynamic Tolling in Tradable Credit Schemes](https://mlsanity.com/paper/2504.08074/) (2025)
- [Temporally smoothed incremental model-based heuristic dynamic programming for command-filtered cascaded online learning flight control](https://mlsanity.com/paper/2507.04346/) (2025)
- [CMR: Contractive Mapping Embeddings for Robust Humanoid Locomotion on Unstructured Terrains](https://mlsanity.com/paper/2602.03511/) (2026)
- [Beyond Coefficients: Forecast-Necessity Testing for Interpretable Causal Discovery in Nonlinear Time-Series Models](https://mlsanity.com/paper/2604.18751/) (2026)
- [Implicit Action Chunking for Smooth Continuous Control](https://mlsanity.com/paper/2605.19592/) (2026)
- [Redesigning Regularization for Effective Policy Smoothing](https://mlsanity.com/paper/2606.13169/) (2026)
- [Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentation](https://mlsanity.com/paper/2609.20300/) (2026)
- [DAMPER: Return-Prioritized Gradient Control for Smooth Policies](https://mlsanity.com/paper/2609.38903/) (2026)

## Abstract (arXiv)

This paper proposes a new regularization technique for reinforcement learning (RL) towards making policy and value functions smooth and stable. RL is known for the instability of the learning process and the sensitivity of the acquired policy to noise. Several methods have been proposed to resolve these problems, and in summary, the smoothness of policy and value functions learned mainly in RL contributes to these problems. However, if these functions are extremely smooth, their expressiveness would be lost, resulting in not obtaining the global optimal solution. This paper therefore considers RL under local Lipschitz continuity constraint, so-called L2C2. By designing the spatio-temporal locally compact space for L2C2 from the state transition at each time step, the moderate smoothness can be achieved without loss of expressiveness. Numerical noisy simulations verified that the proposed L2C2 outperforms the task performance while smoothing out the robot action generated from the learned policy.