# CT-DQN: Control-Tutored Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2212.01343 · HTML: https://mlsanity.com/paper/2212.01343/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 12 of 12)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [A Theoretical Analysis of Deep Q-Learning](https://mlsanity.com/paper/1901.00137/) (2019)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Continuous Deep Q-Learning with Model-based Acceleration](https://mlsanity.com/paper/1603.00748/) (2016)
- [Imitation Learning from Imperfect Demonstration](https://mlsanity.com/paper/1901.09387/) (2019)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Probabilistic design of optimal sequential decision-making algorithms in learning and control](https://mlsanity.com/paper/2201.05212/) (2022)
- [Control-Tutored Reinforcement Learning: Towards the Integration of Data-Driven and Model-Based Control](https://mlsanity.com/paper/2112.06018/) (2021)
- [Driving Reinforcement Learning with Models](https://mlsanity.com/paper/1911.04400/) (2019)
- [Tutoring Reinforcement Learning via Feedback Control](https://mlsanity.com/paper/2012.06863/) (2020)

## Built on by (most cited first, 3 of 3)

- [Guaranteeing Control Requirements via Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2311.10026/) (2023)
- [Data-driven architecture to encode information in the kinematics of robots and artificial avatars](https://mlsanity.com/paper/2403.06557/) (2024)
- [In vivo learning-based control of microbial populations density in bioreactors](https://mlsanity.com/paper/2312.09773/) (2023)

## Abstract (arXiv)

One of the major challenges in Deep Reinforcement Learning for control is the need for extensive training to learn the policy. Motivated by this, we present the design of the Control-Tutored Deep Q-Networks (CT-DQN) algorithm, a Deep Reinforcement Learning algorithm that leverages a control tutor, i.e., an exogenous control law, to reduce learning time. The tutor can be designed using an approximate model of the system, without any assumption about the knowledge of the system's dynamics. There is no expectation that it will be able to achieve the control objective if used stand-alone. During learning, the tutor occasionally suggests an action, thus partially guiding exploration. We validate our approach on three scenarios from OpenAI Gym: the inverted pendulum, lunar lander, and car racing. We demonstrate that CT-DQN is able to achieve better or equivalent data efficiency with respect to the classic function approximation solutions.