mlsanityData to 7 Oct 2026

Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text

arXiv 2005.09382 ↗First posted 19 May 2020cs.CLcited by 59

Not on the map. Why?

Abstract (arXiv)

Recent work has described neural-network-based agents that are trained with reinforcement learning (RL) to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the optimisation of multi-goal motor policies via deep RL from scratch requires many episodes of experience. Consequently, instruction-following with deep RL typically involves language generated from templates (by an environment simulator), which does not reflect the varied or ambiguous expressions of real users. Here, we propose a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions. By applying our method with a state-of-the-art pre-trained text-based language model (BERT), on tasks requiring agents to identify and position everyday objects relative to other objects in a naturalistic 3D simulated room, we demonstrate substantially-above-chance zero-shot transfer from synthetic template commands to natural instructions given by humans. Our approach is a general recipe for training any deep RL-based system to interface with human users, and bridges the gap between two research directions of notable recent success: agent-centric motor behavior and text-based representation learning.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 13; 2022: 12; 2023: 15; 2024: 8; 2025: 7; 2026: 3.

59 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2020Aligning AI With Shared Human Valuescited by 706
  5. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  6. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  7. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  8. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  9. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  10. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  11. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  12. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
38 more
  1. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  2. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  3. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  4. 2025Embodied AI Agents: Modeling the Worldcited by 57
  5. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  6. 2021Language Grounding with 3D Objectscited by 43
  7. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  8. 2021Learning Controllable Content Generatorscited by 40
  9. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  10. 2022Large Language Models can Implement Policy Iterationcited by 31
  11. 2021LILA: Language-Informed Latent Actionscited by 27
  12. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  13. 2021Unsupervised Object-Based Transition Models for 3D Partially Observable Environmentscited by 19
  14. 2023Generalizable Long-Horizon Manipulations with Large Language Modelscited by 16
  15. 2024Object-Centric Instruction Augmentation for Robotic Manipulationcited by 16
  16. 2022Language and Culture Internalisation for Human-Like Autotelic AIcited by 14
  17. 2022Instruction-Following Agents with Multimodal Transformercited by 13
  18. 2021SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmarkcited by 10
  19. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  20. 2023AdaRefiner: Refining Decisions of Language Models with Adaptive Feedbackcited by 10
  21. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedbackcited by 9
  22. 2023Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actionscited by 8
  23. 2023Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learningcited by 8
  24. 2023Improving Long-Horizon Imitation Through Instruction Predictioncited by 8
  25. 2021SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agentscited by 7
  26. 2023Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translationcited by 7
  27. 2022Intra-agent speech permits zero-shot task acquisitioncited by 5
  28. 2024Transformer-based Causal Language Models Perform Clusteringcited by 5
  29. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5
  30. 2021Reinforcement Learning of Implicit and Explicit Control Flow in Instructionscited by 4
  31. 2024A Dialogue Game for Eliciting Balanced Collaborationcited by 4
  32. 2022Learning to Query Internet Text for Informing Reinforcement Learning Agentscited by 3
  33. 2023Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learningcited by 3
  34. 2024Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learningcited by 3
  35. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  36. 2026RoboHarness: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptationcited by 3
  37. 2022Entity Divider with Language Grounding in Multi-Agent Reinforcement Learningcited by 2
  38. 2025Learning Instruction-Following Policies through Open-Ended Instruction Relabeling with Large Language Modelscited by 2

50 most cited of 59 All 59 (JSON)