# Smooth Exploration for Robotic Reinforcement Learning

arXiv: https://arxiv.org/abs/2005.05719 · HTML: https://mlsanity.com/paper/2005.05719/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 30 of 54)

- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Robust Reinforcement Learning from Corrupted Human Feedback](https://mlsanity.com/paper/2406.15568/) (2024)
- [Action Noise in Off-Policy Deep Reinforcement Learning: Impact on Exploration and Performance](https://mlsanity.com/paper/2206.03787/) (2022)
- [Latent Exploration for Reinforcement Learning](https://mlsanity.com/paper/2305.20065/) (2023)
- [Adaptive Preference Scaling for Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2406.02764/) (2024)
- [TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning](https://mlsanity.com/paper/2410.09536/) (2024)
- [SAR: Generalization of Physiological Agility and Dexterity via Synergistic Action Representation](https://mlsanity.com/paper/2307.03716/) (2023)
- [Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns](https://mlsanity.com/paper/2503.03660/) (2025)
- [Testing of Deep Reinforcement Learning Agents with Surrogate Models](https://mlsanity.com/paper/2305.12751/) (2023)
- [Open the Black Box: Step-based Policy Updates for Temporally-Correlated Episodic Reinforcement Learning](https://mlsanity.com/paper/2401.11437/) (2024)
- [Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes](https://mlsanity.com/paper/2210.03512/) (2022)
- [Reinforcement Learning from Simulation to Real World Autonomous Driving using Digital Twin](https://mlsanity.com/paper/2211.14874/) (2022)
- [Emergent Dexterity via Diverse Resets and Large-Scale Reinforcement Learning](https://mlsanity.com/paper/2603.15789/) (2026)
- [Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Sampling](https://mlsanity.com/paper/2312.11091/) (2023)
- [Continual Domain Randomization](https://mlsanity.com/paper/2403.12193/) (2024)
- [DMAP: a Distributed Morphological Attention Policy for Learning to Locomote with a Changing Body](https://mlsanity.com/paper/2209.14218/) (2022)
- [Value-Distributional Model-Based Reinforcement Learning](https://mlsanity.com/paper/2308.06590/) (2023)
- [FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control](https://mlsanity.com/paper/2604.04539/) (2026)
- [Soft Actor-Critic With Integer Actions](https://mlsanity.com/paper/2109.08512/) (2021)
- [Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization](https://mlsanity.com/paper/2312.04386/) (2023)
- [Efficient Reinforcement Learning of Task Planners for Robotic Palletization through Iterative Action Masking Learning](https://mlsanity.com/paper/2404.04772/) (2024)
- [A Multiplicative Value Function for Safe and Efficient Reinforcement Learning](https://mlsanity.com/paper/2303.04118/) (2023)
- [Machine Learning Meets Advanced Robotic Manipulation](https://mlsanity.com/paper/2309.12560/) (2023)
- [Probabilistic Curriculum Learning for Goal-Based Reinforcement Learning](https://mlsanity.com/paper/2504.01459/) (2025)
- [Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2511.10843/) (2025)
- [Zeroth-Order Actor-Critic: An Evolutionary Framework for Sequential Decision Problems](https://mlsanity.com/paper/2201.12518/) (2022)
- [Learning to Exploit Elastic Actuators for Quadruped Locomotion](https://mlsanity.com/paper/2209.07171/) (2022)
- [Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning](https://mlsanity.com/paper/2407.13279/) (2024)
- [Adaptive Data Exploitation in Deep Reinforcement Learning](https://mlsanity.com/paper/2501.12620/) (2025)
- [EvoRL: A GPU-accelerated Framework for Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2501.15129/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) enables robots to learn skills from interactions with the real world. In practice, the unstructured step-based exploration used in Deep RL -- often very successful in simulation -- leads to jerky motion patterns on real robots. Consequences of the resulting shaky behavior are poor exploration, or even damage to the robot. We address these issues by adapting state-dependent exploration (SDE) to current Deep RL algorithms. To enable this adaptation, we propose two extensions to the original SDE, using more general features and re-sampling the noise periodically, which leads to a new exploration method generalized state-dependent exploration (gSDE). We evaluate gSDE both in simulation, on PyBullet continuous control tasks, and directly on three different real robots: a tendon-driven elastic robot, a quadruped and an RC car. The noise sampling interval of gSDE permits to have a compromise between performance and smoothness, which allows training directly on the real robots without loss of performance. The code is available at https://github.com/DLR-RM/stable-baselines3.