mlsanityData to 14 Oct 2026

Redesigning Regularization for Effective Policy Smoothing

arXiv 2606.13169 ↗First posted 11 Jun 2026cs.ROcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper proposes a novel regularization design to effectively smooth policy functions in reinforcement learning. While regularization that enhances ``global'' Lipschitz continuity was initially considered, it has been limited to ``local'' Lipschitz continuity due to a tradeoff between smoothness and expressiveness. However, it has become apparent that the original implementation is cumbersome and does not provide sufficient smoothing, leading to a preference for simpler implementations. This stems from a discrepancy between theory and implementation, and a more appropriate implementation can expect to facilitate smoothing. Therefore, this paper identifies three reasons why the original implementation does not function adequately and provides remedies for them. This modified regularization performs well across multiple tasks and algorithms, successfully achieving smooth motion with sufficient control performance. Furthermore, by applying it to sim-to-real reinforcement learning for a quadruped robot, it is demonstrated that smooth and energy-efficient motion provides robustness against sudden changes in target velocity commands.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 28 of its 42 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

28 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Pseudo-Quantized Actor-Critic Algorithm for Robustness to Noisy Temporal Difference Error2 Apr
  2. 2026Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods30 Jan
  3. 2024Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies22 Oct
  4. 2024Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning5 Jul
  5. 2024On the continuity and smoothness of the value function in reinforcement learning and optimal control21 Mar

Other references · 23

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  3. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  4. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  5. 2020Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Compositioncited by 96
  6. 2020Regularizing Action Policies for Smooth Control with Reinforcement Learningcited by 59
  7. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  8. 2023How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligencecited by 58
  9. 2024Smooth Tchebycheff Scalarization for Multi-Objective Optimizationcited by 54
  10. 2025RSL-RL: A Learning Library for Robotics Researchcited by 49
  11. 2025Learning Humanoid Standing-up Control across Diverse Posturescited by 46
  12. 2021On Tilted Losses in Machine Learning: Theory and Applicationscited by 43
11 more
  1. 2024Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policiescited by 30
  2. 2022L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learningcited by 27
  3. 2025AMOR: Adaptive Character Control through Multi-Objective Reinforcement Learningcited by 9
  4. 2021Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimizationcited by 8
  5. 2025Towards Adaptable Humanoid Control via Adaptive Motion Trackingcited by 8
  6. 2020How to Train your Quadrotor: A Framework for Consistently Smooth and Responsive Flight Control via Reinforcement Learningcited by 7
  7. 2026AGILE: A Comprehensive Workflow for Humanoid Loco-Manipulation Learningcited by 4
  8. 2025Mastering Diverse, Unknown, and Cluttered Tracks for Robust Vision-Based Drone Racingcited by 3
  9. 2026Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding Statescited by 3
  10. 2026Flexible Empowerment at Reasoning with Extended Best-of-N Samplingcited by 3
  11. 2026Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penaltycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting Temporal Regularization for Smooth Control in Deep Reinforcement Learning
  2. 2024Lipschitz-Regularized Critics Lead to Policy Robustness Against Transition Dynamics Uncertainty
  3. 2021Smooth Imitation Learning via Smooth Costs and Smooth Policies
  4. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
  5. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  6. 2025State Entropy Regularization for Robust Reinforcement Learning
  7. 2026Physics-Informed Policy Optimization via Analytic Dynamics Regularization
  8. 2026Higher-Order Action Regularization in Deep Reinforcement Learning: From Continuous Control to Building Energy Management