# Control-Tutored Reinforcement Learning: Towards the Integration of Data-Driven and Model-Based Control

arXiv: https://arxiv.org/abs/2112.06018 · HTML: https://mlsanity.com/paper/2112.06018/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 11 Dec 2021.

## Builds on (most cited first, 12 of 12)

- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Safe Reinforcement Learning Using Robust MPC](https://mlsanity.com/paper/1906.04005/) (2019)
- [Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Map-less Navigation by Leveraging Prior Demonstrations](https://mlsanity.com/paper/1805.07095/) (2018)
- [From self-tuning regulators to reinforcement learning and back again](https://mlsanity.com/paper/1906.11392/) (2019)
- [Safe Reinforcement Learning Using Robust Action Governor](https://mlsanity.com/paper/2102.10643/) (2021)
- [Human-guided Robot Behavior Learning: A GAN-assisted Preference-based Reinforcement Learning Approach](https://mlsanity.com/paper/2010.07467/) (2020)
- [Driving Reinforcement Learning with Models](https://mlsanity.com/paper/1911.04400/) (2019)
- [Tutoring Reinforcement Learning via Feedback Control](https://mlsanity.com/paper/2012.06863/) (2020)

## Built on by (most cited first, 4 of 4)

- [Guaranteeing Control Requirements via Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/2311.10026/) (2023)
- [CT-DQN: Control-Tutored Deep Reinforcement Learning](https://mlsanity.com/paper/2212.01343/) (2022)
- [DR-PETS: Learning-Based Control With Planning in Adversarial Environments](https://mlsanity.com/paper/2503.20660/) (2025)
- [In vivo learning-based control of microbial populations density in bioreactors](https://mlsanity.com/paper/2312.09773/) (2023)

## Abstract (arXiv)

We present an architecture where a feedback controller derived on an approximate model of the environment assists the learning process to enhance its data efficiency. This architecture, which we term as Control-Tutored Q-learning (CTQL), is presented in two alternative flavours. The former is based on defining the reward function so that a Boolean condition can be used to determine when the control tutor policy is adopted, while the latter, termed as probabilistic CTQL (pCTQL), is instead based on executing calls to the tutor with a certain probability during learning. Both approaches are validated, and thoroughly benchmarked against Q-Learning, by considering the stabilization of an inverted pendulum as defined in OpenAI Gym as a representative problem.