# Interactive Learning from Activity Description

arXiv: https://arxiv.org/abs/2102.07024 · HTML: https://mlsanity.com/paper/2102.07024/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 72 references that are in the map.
- First posted on arXiv 13 Feb 2021.

## Builds on (most cited first, 30 of 43)

- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs](https://mlsanity.com/paper/2006.10814/) (2020)
- [Multi-Agent Cooperation and the Emergence of (Natural) Language](https://mlsanity.com/paper/1612.07182/) (2016)
- [Language as an Abstraction for Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07343/) (2019)
- [Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols](https://mlsanity.com/paper/1705.11192/) (2017)
- [Using Natural Language for Reward Shaping in Reinforcement Learning](https://mlsanity.com/paper/1903.02020/) (2019)
- [General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping](https://mlsanity.com/paper/1907.05446/) (2019)
- [Third-Person Imitation Learning](https://mlsanity.com/paper/1703.01703/) (2017)
- [Grounded Language Learning in a Simulated 3D World](https://mlsanity.com/paper/1706.06551/) (2017)
- [Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning](https://mlsanity.com/paper/1909.01871/) (2019)
- [Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog](https://mlsanity.com/paper/1706.08502/) (2017)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Intervention](https://mlsanity.com/paper/1812.04155/) (2018)
- [Learning with Latent Language](https://mlsanity.com/paper/1711.00482/) (2017)
- [Gated-Attention Architectures for Task-Oriented Language Grounding](https://mlsanity.com/paper/1706.07230/) (2017)
- [Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback](https://mlsanity.com/paper/1707.07402/) (2017)
- [Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.06585/) (2017)
- [Reasoning About Pragmatics with Neural Listeners and Speakers](https://mlsanity.com/paper/1604.00562/) (2016)
- [Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration](https://mlsanity.com/paper/2002.09253/) (2020)
- [From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following](https://mlsanity.com/paper/1902.07742/) (2019)
- [Reinforcement Learning with Convex Constraints](https://mlsanity.com/paper/1906.09323/) (2019)
- [Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning](https://mlsanity.com/paper/1805.10627/) (2018)

## Built on by (most cited first, 26 of 26)

- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [Training Language Models with Language Feedback at Scale](https://mlsanity.com/paper/2303.16755/) (2023)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [Aligning LLM Agents by Learning Latent Preference from User Edits](https://mlsanity.com/paper/2404.15269/) (2024)
- [Training Language Models with Language Feedback](https://mlsanity.com/paper/2204.14146/) (2022)
- [LLF-Bench: Benchmark for Interactive Learning from Language Feedback](https://mlsanity.com/paper/2312.06853/) (2023)
- [Distilling Internet-Scale Vision-Language Models into Embodied Agents](https://mlsanity.com/paper/2301.12507/) (2023)
- [Imitating Past Successes can be Very Suboptimal](https://mlsanity.com/paper/2206.03378/) (2022)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Passive learning of active causal strategies in agents and language models](https://mlsanity.com/paper/2305.16183/) (2023)
- [Language and Culture Internalisation for Human-Like Autotelic AI](https://mlsanity.com/paper/2206.01134/) (2022)
- [A Framework for Learning to Request Rich and Contextually Useful Information from Humans](https://mlsanity.com/paper/2110.08258/) (2021)
- [Language-Guided World Models: A Model-Based Approach to AI Control](https://mlsanity.com/paper/2402.01695/) (2024)
- [Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective](https://mlsanity.com/paper/2305.17760/) (2023)
- [Grounding Spatio-Temporal Language with Transformers](https://mlsanity.com/paper/2106.08858/) (2021)
- [SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2107.00956/) (2021)
- [Interaction-Grounded Learning with Action-inclusive Feedback](https://mlsanity.com/paper/2206.08364/) (2022)
- [Learning to Summarize and Answer Questions about a Virtual Robot's Past Actions](https://mlsanity.com/paper/2306.09922/) (2023)
- [Learning to Guide and to Be Guided in the Architect-Builder Problem](https://mlsanity.com/paper/2112.07342/) (2021)
- [Grounding Hindsight Instructions in Multi-Goal Reinforcement Learning for Robotics](https://mlsanity.com/paper/2204.04308/) (2022)
- [Towards Teachable Autotelic Agents](https://mlsanity.com/paper/2105.11977/) (2021)
- [Teachable Reinforcement Learning via Advice Distillation](https://mlsanity.com/paper/2203.11197/) (2022)
- [Progressively Efficient Learning](https://mlsanity.com/paper/2310.13004/) (2023)
- [Provable Interactive Learning with Hindsight Instruction Feedback](https://mlsanity.com/paper/2404.09123/) (2024)

## Abstract (arXiv)

We present a novel interactive learning protocol that enables training request-fulfilling agents by verbally describing their activities. Unlike imitation learning (IL), our protocol allows the teaching agent to provide feedback in a language that is most appropriate for them. Compared with reward in reinforcement learning (RL), the description feedback is richer and allows for improved sample complexity. We develop a probabilistic framework and an algorithm that practically implements our protocol. Empirical results in two challenging request-fulfilling problems demonstrate the strengths of our approach: compared with RL baselines, it is more sample-efficient; compared with IL baselines, it achieves competitive success rates without requiring the teaching agent to be able to demonstrate the desired behavior using the learning agent's actions. Apart from empirical evaluation, we also provide theoretical guarantees for our algorithm under certain assumptions about the teacher and the environment.