mlsanityData to 7 Oct 2026

Continual Learning for Instruction Following from Realtime Feedback

arXiv 2212.09710 ↗First posted 19 Dec 2022cs.CL cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

We propose and deploy an approach to continually train an instruction-following agent from feedback provided by users during collaborative interactions. During interaction, human users instruct an agent using natural language, and provide realtime binary feedback as they observe the agent following their instructions. We design a contextual bandit learning approach, converting user feedback to immediate reward. We evaluate through thousands of human-agent interactions, demonstrating 15.4% absolute improvement in instruction execution accuracy over time. We also show our approach is robust to several design variations, and that the feedback signal is roughly equivalent to the learning signal of supervised demonstration data.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 34 of its 67 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

34 of 67 references.

References · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  10. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  11. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  12. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
22 more
  1. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  2. 2017Learning a Neural Semantic Parser from User Feedbackcited by 98
  3. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  4. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  5. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  6. 2019Executing Instructions in Situated Collaborative Interactionscited by 68
  7. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  8. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  9. 2018Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systemscited by 53
  10. 2016Learning Language Games through Interactioncited by 43
  11. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
  12. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  13. 2019A Natural Language Corpus of Common Grounding under Continuous and Partially-Observable Contextcited by 35
  14. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32
  15. 2019Continual adaptation for efficient machine communicationcited by 28
  16. 2018HexaConvcited by 27
  17. 2018Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learningcited by 27
  18. 2018Learning to Map Context-Dependent Sentences to Executable Formal Queriescited by 23
  19. 2020An Imitation Game for Learning Semantic Parsers from User Interactioncited by 23
  20. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  21. 2022Simulating Bandit Learning from User Feedback for Extractive Question Answeringcited by 14
  22. 2021Analysis of Language Change in Collaborative Instruction Followingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 8; 2025: 3; 2026: 3.

21 papers.

Sort:

By area:

  1. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  2. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  3. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  4. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  5. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  6. 2025Bisecle: Binding and Separation in Continual Learning for Video Language Understandingcited by 13
  7. 2024Grounding Language in Multi-Perspective Referential Communicationcited by 12
  8. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  9. 2023I2I: Initializing Adapters with Improvised Knowledgecited by 7
  10. 2024CoGen: Learning from Feedback with Coupled Comprehension and Generationcited by 7
  11. 2024Retrospective Learning from Interactionscited by 6
  12. 2023Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teacherscited by 2
9 more
  1. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  2. 2023CB2: Collaborative Natural Language Interaction Research Platformcited by 1
  3. 2024Learning Communication Policies for Different Follower Behaviors in a Collaborative Reference Gamecited by 1
  4. 2024Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedbackcited by 1
  5. 2023Design Principles of Robust Multi-Armed Bandit Framework in Video Recommendationscited by 0
  6. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0
  7. 2025Ad hoc conventions generalize to new referentscited by 0
  8. 2026AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injectioncited by 0
  9. 2026Learning to Refer from Estimated Listener Gazecited by 0