# Regularizing Action Policies for Smooth Control with Reinforcement Learning

arXiv: https://arxiv.org/abs/2012.06644 · HTML: https://mlsanity.com/paper/2012.06644/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2020.

## Built on by (most cited first, 30 of 59)

- [L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning](https://mlsanity.com/paper/2202.07152/) (2022)
- [Latent Imagination Facilitates Zero-Shot Transfer in Autonomous Racing](https://mlsanity.com/paper/2103.04909/) (2021)
- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [Benchmarking Offline Reinforcement Learning on Real-Robot Hardware](https://mlsanity.com/paper/2307.15690/) (2023)
- [Delayed Reinforcement Learning by Imitation](https://mlsanity.com/paper/2205.05569/) (2022)
- [Action Tokenizer Matters in In-Context Imitation Learning](https://mlsanity.com/paper/2503.01206/) (2025)
- [Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes](https://mlsanity.com/paper/2210.03512/) (2022)
- [Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning](https://mlsanity.com/paper/2407.04315/) (2024)
- [SkyDreamer: Interpretable End-to-End Vision-Based Drone Racing with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2510.14783/) (2025)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies](https://mlsanity.com/paper/2410.16632/) (2024)
- [Data-Efficient Deep Reinforcement Learning for Attitude Control of Fixed-Wing UAVs: Field Experiments](https://mlsanity.com/paper/2111.04153/) (2021)
- [Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policies](https://mlsanity.com/paper/2301.13019/) (2023)
- [Legged Robot State Estimation With Invariant Extended Kalman Filter Using Neural Measurement Network](https://mlsanity.com/paper/2402.00366/) (2024)
- [AGILE: A Comprehensive Workflow for Humanoid Loco-Manipulation Learning](https://mlsanity.com/paper/2603.20147/) (2026)
- [Policy Regularization for Legible Behavior](https://mlsanity.com/paper/2203.04303/) (2022)
- [Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World](https://mlsanity.com/paper/2308.07741/) (2023)
- [TACO: General Acrobatic Flight Control via Target-and-Command-Oriented Reinforcement Learning](https://mlsanity.com/paper/2503.01125/) (2025)
- [COLSON: Controllable Learning-Based Social Navigation via Diffusion-Based Reinforcement Learning](https://mlsanity.com/paper/2503.13934/) (2025)
- [Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization](https://mlsanity.com/paper/2511.15055/) (2025)
- [Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States](https://mlsanity.com/paper/2601.18479/) (2026)
- [Arnold: A multi-task, multi-embodiment muscle transformer policy](https://mlsanity.com/paper/2508.18066/) (2025)
- [Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods](https://mlsanity.com/paper/2601.22970/) (2026)
- [Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty](https://mlsanity.com/paper/2602.18312/) (2026)
- [Decentralized Global Connectivity Maintenance for Multi-Robot Navigation: A Reinforcement Learning Approach](https://mlsanity.com/paper/2109.08536/) (2021)
- [Reinforcement Learning-Based Air Traffic Deconfliction](https://mlsanity.com/paper/2301.01861/) (2023)
- [Sense, Imagine, Act: Multimodal Perception Improves Model-Based Reinforcement Learning for Head-to-Head Autonomous Racing](https://mlsanity.com/paper/2305.04750/) (2023)
- [Towards Transferring Tactile-based Continuous Force Control Policies from Simulation to Robot](https://mlsanity.com/paper/2311.07245/) (2023)
- [Partial End-to-end Reinforcement Learning for Robustness Against Modelling Error in Autonomous Racing](https://mlsanity.com/paper/2312.06406/) (2023)
- [Explainable AI for Ship Collision Avoidance: Decoding Decision-Making Processes and Behavioral Intentions](https://mlsanity.com/paper/2405.09081/) (2024)

## Abstract (arXiv)

A critical problem with the practical utility of controllers trained with deep Reinforcement Learning (RL) is the notable lack of smoothness in the actions learned by the RL policies. This trend often presents itself in the form of control signal oscillation and can result in poor control, high power consumption, and undue system wear. We introduce Conditioning for Action Policy Smoothness (CAPS), an effective yet intuitive regularization on action policies, which offers consistent improvement in the smoothness of the learned state-to-action mappings of neural network controllers, reflected in the elimination of high-frequency components in the control signal. Tested on a real system, improvements in controller smoothness on a quadrotor drone resulted in an almost 80% reduction in power consumption while consistently training flight-worthy controllers. Project website: http://ai.bu.edu/caps