# Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1812.05027 · HTML: https://mlsanity.com/paper/1812.05027/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2018.

## Built on by (most cited first, 17 of 17)

- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [LazyDAgger: Reducing Context Switching in Interactive Imitation Learning](https://mlsanity.com/paper/2104.00053/) (2021)
- [Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervision](https://mlsanity.com/paper/2206.14349/) (2022)
- [Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics](https://mlsanity.com/paper/2211.02231/) (2022)
- [Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics](https://mlsanity.com/paper/2107.09822/) (2021)
- [Curriculum Learning for Safe Mapless Navigation](https://mlsanity.com/paper/2112.12490/) (2021)
- [IPAPRec: A promising tool for learning high-performance mapless navigation skills with deep reinforcement learning](https://mlsanity.com/paper/2103.11686/) (2021)
- [Handling Sparse Rewards in Reinforcement Learning Using Model Predictive Control](https://mlsanity.com/paper/2210.01525/) (2022)
- [Using Deep Reinforcement Learning with Automatic Curriculum Learning for Mapless Navigation in Intralogistics](https://mlsanity.com/paper/2202.11512/) (2022)
- [Combining RL and IL using a dynamic, performance-based modulation over learning signals and its application to local planning](https://mlsanity.com/paper/2405.09760/) (2024)
- [Is FISHER All You Need in The Multi-AUV Underwater Target Tracking Task?](https://mlsanity.com/paper/2412.03959/) (2024)
- [Job Scheduling in Datacenters using Constraint Controlled RL](https://mlsanity.com/paper/2211.05338/) (2022)
- [Enhancing Deep Reinforcement Learning-based Robot Navigation Generalization through Scenario Augmentation](https://mlsanity.com/paper/2503.01146/) (2025)
- [Accelerate Reinforcement Learning with PID Controllers in the Pendulum Simulations](https://mlsanity.com/paper/2210.00770/) (2022)
- [Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations](https://mlsanity.com/paper/2509.15981/) (2025)
- [Safe Online Learning via Smooth Safety-Structured Policy Composition](https://mlsanity.com/paper/2606.31320/) (2026)
- [Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning](https://mlsanity.com/paper/2610.06019/) (2026)

## Abstract (arXiv)

Deep Reinforcement Learning (DRL) has been applied successfully to many robotic applications. However, the large number of trials needed for training is a key issue. Most of existing techniques developed to improve training efficiency (e.g. imitation) target on general tasks rather than being tailored for robot applications, which have their specific context to benefit from. We propose a novel framework, Assisted Reinforcement Learning, where a classical controller (e.g. a PID controller) is used as an alternative, switchable policy to speed up training of DRL for local planning and navigation problems. The core idea is that the simple control law allows the robot to rapidly learn sensible primitives, like driving in a straight line, instead of random exploration. As the actor network becomes more advanced, it can then take over to perform more complex actions, like obstacle avoidance. Eventually, the simple controller can be discarded entirely. We show that not only does this technique train faster, it also is less sensitive to the structure of the DRL network and consistently outperforms a standard Deep Deterministic Policy Gradient network. We demonstrate the results in both simulation and real-world experiments.