# Continual Learning for Instruction Following from Realtime Feedback

arXiv: https://arxiv.org/abs/2212.09710 · HTML: https://mlsanity.com/paper/2212.09710/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 67 references that are in the map.
- First posted on arXiv 19 Dec 2022.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction](https://mlsanity.com/paper/1809.00786/) (2018)
- [Learning a Neural Semantic Parser from User Feedback](https://mlsanity.com/paper/1704.08760/) (2017)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback](https://mlsanity.com/paper/1707.07402/) (2017)
- [Can Neural Machine Translation be Improved with User Feedback?](https://mlsanity.com/paper/1804.05958/) (2018)
- [Executing Instructions in Situated Collaborative Interactions](https://mlsanity.com/paper/1910.03655/) (2019)
- [Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning](https://mlsanity.com/paper/1805.10627/) (2018)
- [Deep Reinforcement Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1902.04257/) (2019)
- [Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1804.06512/) (2018)
- [Learning Language Games through Interaction](https://mlsanity.com/paper/1606.02447/) (2016)
- [The EMPATHIC Framework for Task Learning from Implicit Human Feedback](https://mlsanity.com/paper/2009.13649/) (2020)
- [Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction](https://mlsanity.com/paper/1811.04179/) (2018)
- [A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Context](https://mlsanity.com/paper/1907.03399/) (2019)
- [Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback](https://mlsanity.com/paper/1805.01252/) (2018)
- [Continual adaptation for efficient machine communication](https://mlsanity.com/paper/1911.09896/) (2019)
- [HexaConv](https://mlsanity.com/paper/1803.02108/) (2018)
- [Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learning](https://mlsanity.com/paper/1806.00047/) (2018)
- [Learning to Map Context-Dependent Sentences to Executable Formal Queries](https://mlsanity.com/paper/1804.06868/) (2018)

## Built on by (most cited first, 22 of 22)

- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2311.00168/) (2023)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [MLLM-CL: Continual Learning for Multimodal Large Language Models](https://mlsanity.com/paper/2506.05453/) (2025)
- [Bisecle: Binding and Separation in Continual Learning for Video Language Understanding](https://mlsanity.com/paper/2507.00469/) (2025)
- [Grounding Language in Multi-Perspective Referential Communication](https://mlsanity.com/paper/2410.03959/) (2024)
- [Continually Improving Extractive QA via Human Feedback](https://mlsanity.com/paper/2305.12473/) (2023)
- [I2I: Initializing Adapters with Improvised Knowledge](https://mlsanity.com/paper/2304.02168/) (2023)
- [CoGen: Learning from Feedback with Coupled Comprehension and Generation](https://mlsanity.com/paper/2408.15992/) (2024)
- [Retrospective Learning from Interactions](https://mlsanity.com/paper/2410.13852/) (2024)
- [Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teachers](https://mlsanity.com/paper/2305.12880/) (2023)
- [φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models](https://mlsanity.com/paper/2602.22601/) (2026)
- [CB2: Collaborative Natural Language Interaction Research Platform](https://mlsanity.com/paper/2303.08127/) (2023)
- [Learning Communication Policies for Different Follower Behaviors in a Collaborative Reference Game](https://mlsanity.com/paper/2402.04824/) (2024)
- [Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback](https://mlsanity.com/paper/2403.11330/) (2024)
- [AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection](https://mlsanity.com/paper/2608.04053/) (2026)
- [Design Principles of Robust Multi-Armed Bandit Framework in Video Recommendations](https://mlsanity.com/paper/2310.01419/) (2023)
- [Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Model](https://mlsanity.com/paper/2410.16006/) (2024)
- [Ad hoc conventions generalize to new referents](https://mlsanity.com/paper/2509.05566/) (2025)
- [Learning to Refer from Estimated Listener Gaze](https://mlsanity.com/paper/2609.14207/) (2026)
- [Agent-Reflex: Teaching Language Agents to Act Reflectively](https://mlsanity.com/paper/2610.17953/) (2026)

## Abstract (arXiv)

We propose and deploy an approach to continually train an instruction-following agent from feedback provided by users during collaborative interactions. During interaction, human users instruct an agent using natural language, and provide realtime binary feedback as they observe the agent following their instructions. We design a contextual bandit learning approach, converting user feedback to immediate reward. We evaluate through thousands of human-agent interactions, demonstrating 15.4% absolute improvement in instruction execution accuracy over time. We also show our approach is robust to several design variations, and that the feedback signal is roughly equivalent to the learning signal of supervised demonstration data.