mlsanityData to 7 Oct 2026

Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

arXiv 1704.08795 ↗First posted 28 Apr 2017cs.CLcited by 80

Not on the map. Why?

Abstract (arXiv)

We propose to directly map raw visual observations and text input to actions for instruction execution. While existing approaches assume access to structured environment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about linguistic and visual input. We use reinforcement learning in a contextual bandit setting to train a neural network agent. To guide the agent's exploration, we use reward shaping with different forms of supervision. Our approach does not require intermediate representations, planning procedures, or training different models. We evaluate in a simulated environment, and show significant improvements over supervised learning and common reinforcement learning variants.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 2; 2019: 3; 2020: 1; 2021: 18; 2022: 14; 2023: 24; 2024: 13; 2025: 3; 2026: 2.

80 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  3. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  4. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  5. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  6. 2023Language-Driven Representation Learning for Roboticscited by 163
  7. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  8. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  9. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  10. 2021Skill Induction and Planning with Latent Languagecited by 99
  11. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  12. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
38 more
  1. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  2. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  3. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  4. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  5. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  6. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 55
  7. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  8. 2022Inferring Rewards from Language in Contextcited by 50
  9. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  10. 2023Learning to Generate Better Than Your LLMcited by 49
  11. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  12. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  13. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  14. 2021Grounding Open-Domain Instructions to Automate Web Support Taskscited by 43
  15. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  16. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  17. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  18. 2023Learning adaptive planning representations with natural language guidancecited by 36
  19. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  20. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  21. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  22. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  23. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  24. 2022Learning to Execute Actions or Ask Clarification Questionscited by 30
  25. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  26. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  27. 2018Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learningcited by 27
  28. 2021Interactive Learning from Activity Descriptioncited by 26
  29. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulationcited by 26
  30. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  31. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  32. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  33. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  34. 2021Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Visioncited by 18
  35. 2021Who's Waldo? Linking People Across Text and Imagescited by 16
  36. 2024CoPESD: A Multi-Level Surgical Motion Dataset for Training Large Vision-Language Models to Co-Pilot Endoscopic Submucosal Dissectioncited by 15
  37. 2022MMKGR: Multi-hop Multi-modal Knowledge Graph Reasoningcited by 13
  38. 2024Language-guided Skill Learning with Temporal Variational Inferencecited by 11

50 most cited of 80 All 80 (JSON)