# Towards Teachable Autotelic Agents

arXiv: https://arxiv.org/abs/2105.11977 · HTML: https://mlsanity.com/paper/2105.11977/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 139 references that are in the map.
- First posted on arXiv 25 May 2021.

## Builds on (most cited first, 28 of 28)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [A Survey of Reinforcement Learning Informed by Natural Language](https://mlsanity.com/paper/1906.03926/) (2019)
- [Automatic Curriculum Learning For Deep RL: A Short Survey](https://mlsanity.com/paper/2003.04664/) (2020)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning](https://mlsanity.com/paper/1708.02190/) (2017)
- [Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration](https://mlsanity.com/paper/2002.09253/) (2020)
- [Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning](https://mlsanity.com/paper/1912.11032/) (2019)
- [Learning Human Objectives by Evaluating Hypothetical Behavior](https://mlsanity.com/paper/1912.05652/) (2019)
- [Contextual Imagined Goals for Self-Supervised Robotic Learning](https://mlsanity.com/paper/1910.11670/) (2019)
- [Automated curricula through setter-solver interactions](https://mlsanity.com/paper/1909.12892/) (2019)
- [Interactive Learning from Activity Description](https://mlsanity.com/paper/2102.07024/) (2021)
- [Feature Expansive Reward Learning: Rethinking Human Input](https://mlsanity.com/paper/2006.13208/) (2020)
- [Symbolic Behaviour in Artificial Intelligence](https://mlsanity.com/paper/2102.03406/) (2021)
- [Inverse Reinforcement Learning with Natural Language Goals](https://mlsanity.com/paper/2008.06924/) (2020)
- [Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems](https://mlsanity.com/paper/2007.01195/) (2020)
- [Curiosity-Driven Multi-Criteria Hindsight Experience Replay](https://mlsanity.com/paper/1906.03710/) (2019)
- [CLIC: Curriculum Learning and Imitation for object Control in non-rewarding environments](https://mlsanity.com/paper/1901.09720/) (2019)
- [AI Challenges in Human-Robot Cognitive Teaming](https://mlsanity.com/paper/1707.04775/) (2017)
- [ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/1910.12154/) (2019)
- [Theory of Mind with Guilt Aversion Facilitates Cooperative Reinforcement Learning](https://mlsanity.com/paper/2009.07445/) (2020)
- [Learning Compositional Neural Programs for Continuous Control](https://mlsanity.com/paper/2007.13363/) (2020)
- [Explainable Agents Through Social Cues: A Review](https://mlsanity.com/paper/2003.05251/) (2020)

## Built on by (most cited first, 2 of 2)

- [From Interactive to Co-Constructive Task Learning](https://mlsanity.com/paper/2305.15535/) (2023)
- [Utility-based Adaptive Teaching Strategies using Bayesian Theory of Mind](https://mlsanity.com/paper/2309.17275/) (2023)

## Abstract (arXiv)

Autonomous discovery and direct instruction are two distinct sources of learning in children but education sciences demonstrate that mixed approaches such as assisted discovery or guided play result in improved skill acquisition. In the field of Artificial Intelligence, these extremes respectively map to autonomous agents learning from their own signals and interactive learning agents fully taught by their teachers. In between should stand teachable autotelic agents (TAA): agents that learn from both internal and teaching signals to benefit from the higher efficiency of assisted discovery. Designing such agents will enable real-world non-expert users to orient the learning trajectories of agents towards their expectations. More fundamentally, this may also be a key step to build agents with human-level intelligence. This paper presents a roadmap towards the design of teachable autonomous agents. Building on developmental psychology and education sciences, we start by identifying key features enabling assisted discovery processes in child-tutor interactions. This leads to the production of a checklist of features that future TAA will need to demonstrate. The checklist allows us to precisely pinpoint the various limitations of current reinforcement learning agents and to identify the promising first steps towards TAA. It also shows the way forward by highlighting key research directions towards the design or autonomous agents that can be taught by ordinary people via natural pedagogy.