mlsanityData to 14 Oct 2026

Episodic Transformer for Vision-and-Language Navigation

arXiv 2105.06453 ↗First posted 13 May 2021cs.CV cs.AIcited by 156

Not on the map. Why?

Abstract (arXiv)

Interaction and navigation defined by natural language instructions in dynamic environments pose significant challenges for neural agents. This paper focuses on addressing two challenges: handling long sequence of subtasks, and understanding complex human instructions. We propose Episodic Transformer (E.T.), a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions. To improve training, we leverage synthetic instructions as an intermediate representation that decouples understanding the visual appearance of an environment from the variations of natural language instructions. We demonstrate that encoding the history with a transformer is critical to solve compositional tasks, and that pretraining and joint training with synthetic instructions further improve the performance. Our approach sets a new state of the art on the challenging ALFRED benchmark, achieving 38.4% and 8.5% task success rates on seen and unseen test splits.

Key facts. Cited by 156 papers in the map (data through 14 Oct 2026). Builds on 47 of its 73 references that are in the map. First posted on arXiv 13 May 2021.

Builds on

47 of 73 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2017Mask R-CNNcited by 7,277
  9. 2017Non-local Neural Networkscited by 2,293
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  12. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
35 more
  1. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  3. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  4. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  5. 2020Quantifying Attention Flow in Transformerscited by 775
  6. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  7. 2018On Evaluation of Embodied Navigation Agentscited by 547
  8. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  9. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  10. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  11. 2020Multi-modal Transformer for Video Retrievalcited by 395
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  13. 2018Video Action Transformer Networkcited by 316
  14. 2019Learning Latent Plans from Playcited by 316
  15. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  16. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  17. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  18. 2019Causal Confusion in Imitation Learningcited by 288
  19. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  20. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  21. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  22. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 193
  23. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  24. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  25. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  26. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 105
  27. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 81
  28. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  29. 2014Grammar as a Foreign Languagecited by 70
  30. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  31. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  32. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 48
  33. 2019Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigationcited by 40
  34. 2015Alignment-based compositional semantics for instruction followingcited by 21
  35. 2020Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street Viewcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 42; 2023: 49; 2024: 32; 2025: 10; 2026: 13.

156 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  4. 2022Multimodal Learning with Transformers: A Surveycited by 400
  5. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  6. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  7. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  8. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  9. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  10. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  11. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  12. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
38 more
  1. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  2. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 124
  3. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  4. 2021FILM: Following Instructions in Language with Modular Methodscited by 121
  5. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  6. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 110
  7. 2021Skill Induction and Planning with Latent Languagecited by 99
  8. 2022Instruction-driven history-aware policies for robotic manipulationscited by 94
  9. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  10. 2022Cross-modal Map Learning for Vision and Language Navigationcited by 91
  11. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  12. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 85
  13. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  14. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 72
  15. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 69
  16. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  17. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 67
  18. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 65
  19. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 64
  20. 2022Video Transformers: A Surveycited by 63
  21. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  22. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  23. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 58
  24. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  25. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  26. 2022P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervisioncited by 52
  27. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 50
  28. 2022Aerial Vision-and-Dialog Navigationcited by 49
  29. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  30. 2022Retrospectives on the Embodied AI Workshopcited by 46
  31. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  32. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 43
  33. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 40
  34. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  35. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  36. 20221st Place Solutions for RxR-Habitat Vision-and-Language Navigation Competition (CVPR 2022)cited by 34
  37. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  38. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33

50 most cited of 156 All 156 (JSON)