mlsanityData to 7 Oct 2026

FILM: Following Instructions in Language with Modular Methods

arXiv 2110.07342 ↗First posted 12 Oct 2021cs.CL cs.LGcited by 119

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Recent methods for embodied instruction following are typically trained end-to-end using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume that neural states will integrate multimodal semantics to perform state tracking, building spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46 %) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49 %). Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or low-level instructions.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 43 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

31 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation25 Nov

Other references · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017Mask R-CNNcited by 7,259
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  9. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  10. 2018On Evaluation of Embodied Navigation Agentscited by 526
  11. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
18 more
  1. 2017Embodied Question Answeringcited by 335
  2. 2020Learning to Explore using Active Neural SLAMcited by 308
  3. 2020Experience Grounds Languagecited by 241
  4. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  5. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  6. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  7. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  8. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  9. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  10. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  11. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  12. 2021VTNet: Visual Transformer Network for Object Goal Navigationcited by 50
  13. 2017A-Fast-RCNN: Hard Positive Generation via Adversary for Object Detectioncited by 43
  14. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  15. 2018Following High-level Navigation Instructions on a Simulated Quadcopter with Imitation Learningcited by 27
  16. 2021Look Wide and Interpret Twice: Improving Performance on Interactive Instruction-following Taskscited by 25
  17. 2019Embodied Language Grounding with 3D Visual Feature Representationscited by 20
  18. 2020Language-Mediated, Object-Centric Representation Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 28; 2023: 34; 2024: 34; 2025: 15; 2026: 8.

119 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  3. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  4. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  5. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  6. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  7. 2023ADaPT: As-Needed Decomposition and Planning with Language Modelscited by 172
  8. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  9. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  10. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  11. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  12. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
38 more
  1. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  2. 2022Navigating to Objects in the Real Worldcited by 104
  3. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  4. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  5. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  6. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  7. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  8. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  9. 2022Retrospectives on the Embodied AI Workshopcited by 46
  10. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  11. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  12. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  13. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 42
  14. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  15. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  16. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  17. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  18. 2023Context-Aware Planning and Environment-Aware Memory for Instruction Following Embodied Agentscited by 33
  19. 2024OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Roboticscited by 33
  20. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  21. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  22. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  23. 2024EPO: Hierarchical LLM Agents with Environment Preference Optimizationcited by 28
  24. 2022One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestonescited by 26
  25. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  26. 2024Tools Fail: Detecting Silent Errors in Faulty Toolscited by 23
  27. 2022Few-shot Subgoal Planning with Language Modelscited by 22
  28. 2024Online Continual Learning For Interactive Instruction Following Agentscited by 21
  29. 2023ThinkBot: Embodied Instruction Following with Thought Chain Reasoningcited by 19
  30. 2023Alexa Arena: A User-Centric Interactive Platform for Embodied AIcited by 18
  31. 2024Zero-shot Object Navigation with Vision-Language Models Reasoningcited by 18
  32. 2024ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environmentscited by 16
  33. 2022DANLI: Deliberative Agent for Following Natural Language Instructionscited by 15
  34. 2024OPEx: A Component-Wise Analysis of LLM-Centric Agents in Embodied Instruction Followingcited by 14
  35. 2022Finding Fallen Objects Via Asynchronous Audio-Visual Integrationcited by 13
  36. 2022Dialog Acts for Task-Driven Embodied Agentscited by 13
  37. 2023Reasoning about the Unseen for Efficient Outdoor Object Navigationcited by 13
  38. 2023MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videoscited by 12

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026RoboFollow: Unveiling the Instruction Following Mirage in Embodied Agents
  2. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control
  3. 2026InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation
  4. 2024Semantic Skill Grounding for Embodied Instruction-Following in Cross-Domain Environments
  5. 2024Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following
  6. 2026Empowering Reliable Visual-Centric Instruction Following in MLLMs
  7. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs
  8. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuning