mlsanityData to 7 Oct 2026

Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences

arXiv 1506.04089 ↗First posted 12 Jun 2015cs.CL cs.AI cs.LGcited by 50

Not on the map. Why?

Abstract (arXiv)

We propose a neural sequence-to-sequence model for direction following, a task that is essential to realizing effective autonomous agents. Our alignment-based encoder-decoder model with long short-term memory recurrent neural networks (LSTM-RNN) translates natural language instructions to action sequences based upon a representation of the observable world state. We introduce a multi-level aligner that empowers our model to focus on sentence "regions" salient to the current world state by using multiple abstractions of the input sentence. In contrast to existing methods, our model uses no specialized linguistic resources (e.g., parsers) or task-specific annotations (e.g., seed lexicons). It is therefore generalizable, yet still achieves the best results reported to-date on a benchmark single-sentence dataset and competitive results for the limited-training multi-sentence setting. We analyze our model through a series of ablations that elucidate the contributions of the primary components of our model.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2015.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2018: 1; 2019: 1; 2020: 1; 2021: 14; 2022: 12; 2023: 11; 2024: 8; 2025: 2.

50 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  4. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  5. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  6. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  7. 2023Large Language Models for Robotics: A Surveycited by 132
  8. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  9. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  10. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  11. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  12. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
38 more
  1. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  2. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  3. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  4. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  5. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  6. 2022Inferring Rewards from Language in Contextcited by 50
  7. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  8. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  9. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  10. 2021Diagnosing Vision-and-Language Navigation: What Really Matterscited by 36
  11. 2024The broader spectrum of in-context learningcited by 33
  12. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  13. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  14. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  15. 2021Interactive Learning from Activity Descriptioncited by 26
  16. 2021GPT3-to-plan: Extracting plans from text using GPT-3cited by 26
  17. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  18. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  19. 2022Few-shot Subgoal Planning with Language Modelscited by 22
  20. 2021Towards Navigation by Reasoning over Spatial Configurationscited by 14
  21. 2024MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Modelscited by 9
  22. 2023Conceptual Reinforcement Learning for Language-Conditioned Taskscited by 8
  23. 2021A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environmentcited by 6
  24. 2021Language Understanding for Field and Service Robots in a Priori Unknown Environmentscited by 6
  25. 2020Combating False Negatives in Adversarial Imitation Learningcited by 4
  26. 2024LIMT: Language-Informed Multi-Task Visual World Modelscited by 4
  27. 2022Integrating AI Planning with Natural Language Processing: A Combination of Explicit and Tacit Knowledgecited by 3
  28. 2023Automated Action Model Acquisition from Narrative Textscited by 3
  29. 2024Learning Planning Abstractions from Languagecited by 3
  30. 2022Teachable Reinforcement Learning via Advice Distillationcited by 2
  31. 2023Learning to Play Chess from Textbooks (LEAP): a Corpus for Evaluating Chess Moves based on Sentiment Analysiscited by 2
  32. 2021Guided Policy Search for Parameterized Skills using Adverbscited by 1
  33. 2024Provable Interactive Learning with Hindsight Instruction Feedbackcited by 1
  34. 2024Aspect-based Sentiment Evaluation of Chess Moves (ASSESS): an NLP-based Method for Evaluating Chess Strategies from Textbookscited by 1
  35. 2025Sample-Efficient Optimization over Generative Priors via Coarse Learnabilitycited by 1
  36. 2021Grounding Complex Navigational Instructions Using Scene Graphscited by 0
  37. 2022Learning Invariable Semantical Representation from Language for Extensible Policy Generalizationcited by 0
  38. 2024LASMP: Language Aided Subset Sampling Based Motion Plannercited by 0