# Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States

arXiv: https://arxiv.org/abs/2601.18479 · HTML: https://mlsanity.com/paper/2601.18479/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 25 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective](https://mlsanity.com/paper/2105.05246/) (2021)
- [Regularizing Action Policies for Smooth Control with Reinforcement Learning](https://mlsanity.com/paper/2012.06644/) (2020)
- [L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning](https://mlsanity.com/paper/2202.07152/) (2022)
- [Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning](https://mlsanity.com/paper/2407.04315/) (2024)
- [Addressing Action Oscillations through Learning Policy Inertia](https://mlsanity.com/paper/2103.02287/) (2021)
- [Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies](https://mlsanity.com/paper/2410.16632/) (2024)

## Built on by (most cited first, 3 of 3)

- [Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods](https://mlsanity.com/paper/2601.22970/) (2026)
- [Redesigning Regularization for Effective Policy Smoothing](https://mlsanity.com/paper/2606.13169/) (2026)
- [DAMPER: Return-Prioritized Gradient Control for Smooth Policies](https://mlsanity.com/paper/2609.38903/) (2026)

## Abstract (arXiv)

Deep reinforcement learning has proven to be a powerful approach to solving control tasks, but its characteristic high-frequency oscillations make it difficult to apply in real-world environments. While prior methods have addressed action oscillations via architectural or loss-based methods, the latter typically depend on heuristic or synthetic definitions of state similarity to promote action consistency, which often fail to accurately reflect the underlying system dynamics. In this paper, we propose a novel loss-based method by introducing a transition-induced similar state. The transition-induced similar state is defined as the distribution of next states transitioned from the previous state. Since it utilizes only environmental feedback and actually collected data, it better captures system dynamics. Building upon this foundation, we introduce Action Smoothing by Aligning Actions with Predictions from Preceding States (ASAP), an action smoothing method that effectively mitigates action oscillations. ASAP enforces action smoothness by aligning the actions with those taken in transition-induced similar states and by penalizing second-order differences to suppress high-frequency oscillations. Experiments in Gymnasium and Isaac-Lab environments demonstrate that ASAP yields smoother control and improved policy performance over existing methods.