# Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior

arXiv: https://arxiv.org/abs/2108.04812 · HTML: https://mlsanity.com/paper/2108.04812/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 10 Aug 2021.

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Deal or No Deal? End-to-End Learning for Negotiation Dialogues](https://mlsanity.com/paper/1706.05125/) (2017)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Optimal and Adaptive Off-policy Evaluation in Contextual Bandits](https://mlsanity.com/paper/1612.01205/) (2016)
- [Human-centric Dialog Training via Offline Reinforcement Learning](https://mlsanity.com/paper/2010.05848/) (2020)
- [Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback](https://mlsanity.com/paper/1707.07402/) (2017)
- [Can Neural Machine Translation be Improved with User Feedback?](https://mlsanity.com/paper/1804.05958/) (2018)
- [Executing Instructions in Situated Collaborative Interactions](https://mlsanity.com/paper/1910.03655/) (2019)
- [Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning](https://mlsanity.com/paper/1805.10627/) (2018)
- [Deep Reinforcement Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1902.04257/) (2019)
- [Unified Pragmatic Models for Generating and Following Instructions](https://mlsanity.com/paper/1711.04987/) (2017)
- [Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1804.06512/) (2018)
- [On the Evaluation of Vision-and-Language Navigation Instructions](https://mlsanity.com/paper/2101.10504/) (2021)
- [Learning Language Games through Interaction](https://mlsanity.com/paper/1606.02447/) (2016)
- [Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight](https://mlsanity.com/paper/1910.09664/) (2019)
- [Characterizing the dynamics of learning in repeated reference games](https://mlsanity.com/paper/1912.07199/) (2019)
- [Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback](https://mlsanity.com/paper/1805.01252/) (2018)
- [Bandit Structured Prediction for Neural Sequence-to-Sequence Learning](https://mlsanity.com/paper/1704.06497/) (2017)
- [Continual adaptation for efficient machine communication](https://mlsanity.com/paper/1911.09896/) (2019)
- [HexaConv](https://mlsanity.com/paper/1803.02108/) (2018)
- [Encoder-Agnostic Adaptation for Conditional Language Generation](https://mlsanity.com/paper/1908.06938/) (2019)
- [Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translation](https://mlsanity.com/paper/1707.09118/) (2017)

## Built on by (most cited first, 19 of 19)

- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [Less is More: Generating Grounded Navigation Instructions from Landmarks](https://mlsanity.com/paper/2111.12872/) (2021)
- [Continual Learning for Instruction Following from Realtime Feedback](https://mlsanity.com/paper/2212.09710/) (2022)
- [Simulating Bandit Learning from User Feedback for Extractive Question Answering](https://mlsanity.com/paper/2203.10079/) (2022)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [Grounding Language in Multi-Perspective Referential Communication](https://mlsanity.com/paper/2410.03959/) (2024)
- [Continually Improving Extractive QA via Human Feedback](https://mlsanity.com/paper/2305.12473/) (2023)
- [Collecting Interactive Multi-modal Datasets for Grounded Language Understanding](https://mlsanity.com/paper/2211.06552/) (2022)
- [Infer Human's Intentions Before Following Natural Language Instructions](https://mlsanity.com/paper/2409.18073/) (2024)
- [MUG: Interactive Multimodal Grounding on User Interfaces](https://mlsanity.com/paper/2209.15099/) (2022)
- [Dealing with Semantic Underspecification in Multimodal NLP](https://mlsanity.com/paper/2306.05240/) (2023)
- [CoGen: Learning from Feedback with Coupled Comprehension and Generation](https://mlsanity.com/paper/2408.15992/) (2024)
- [Retrospective Learning from Interactions](https://mlsanity.com/paper/2410.13852/) (2024)
- [Many Episode Learning in a Modular Embodied Agent via End-to-End Interaction](https://mlsanity.com/paper/2204.08687/) (2022)
- [CB2: Collaborative Natural Language Interaction Research Platform](https://mlsanity.com/paper/2303.08127/) (2023)
- [Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback](https://mlsanity.com/paper/2403.11330/) (2024)
- [Mixed-Initiative Dialog for Human-Robot Collaborative Manipulation](https://mlsanity.com/paper/2508.05535/) (2025)
- [Efficient Test-Time Adaptation through Human-AI Interaction](https://mlsanity.com/paper/2609.04141/) (2026)
- [Learning to Refer from Estimated Listener Gaze](https://mlsanity.com/paper/2609.14207/) (2026)

## Abstract (arXiv)

We study continual learning for natural language instruction generation, by observing human users' instruction execution. We focus on a collaborative scenario, where the system both acts and delegates tasks to human users using natural language. We compare user execution of generated instructions to the original system intent as an indication to the system's success communicating its intent. We show how to use this signal to improve the system's ability to generate instructions via contextual bandit learning. In interaction with real users, our system demonstrates dramatic improvements in its ability to generate language over time.