mlsanityData to 7 Oct 2026

History Aware Multimodal Transformer for Vision-and-Language Navigation

arXiv 2110.13309 ↗First posted 25 Oct 2021cs.CV cs.AIcited by 268

Not on the map. Why?

Abstract (arXiv)

Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches to VLN implement memory using recurrent states. Instead, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy. HAMT achieves new state of the art on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN (R4R, R2R-Back). We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories.

Key facts. Cited by 268 papers in the map (data through 7 Oct 2026). Builds on 52 of its 62 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

52 of 62 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2016Layer Normalizationcited by 5,579
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2021ViViT: A Video Vision Transformercited by 1,610
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
38 more
  1. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  5. 2016Deep Networks with Stochastic Depthcited by 973
  6. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  8. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  9. 2018On Evaluation of Embodied Navigation Agentscited by 526
  10. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  11. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  12. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  13. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  14. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  15. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  16. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  17. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  18. 2019Vision-and-Dialog Navigationcited by 246
  19. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  20. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  21. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  22. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  23. 2018Semi-parametric Topological Memory for Navigationcited by 203
  24. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  25. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  26. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  27. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  28. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  29. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  30. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  31. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  32. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  33. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  34. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  35. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  36. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  37. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  38. 2020Diagnosing the Environment Bias in Vision-and-Language Navigationcited by 41

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 28; 2023: 52; 2024: 60; 2025: 59; 2026: 66.

268 papers.

Sort:

By area:

  1. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  4. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  5. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  6. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  7. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  8. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  9. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  10. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  11. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  12. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
38 more
  1. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  2. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
  3. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  4. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  5. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  6. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  7. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  8. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  9. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  10. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  11. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 80
  12. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  13. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  14. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  15. 2023A Comprehensive Survey on Deep Graph Representation Learningcited by 72
  16. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 71
  17. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  18. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  19. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  20. 2024Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigationcited by 63
  21. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  22. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  23. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  24. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 60
  25. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  26. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  27. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  28. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  29. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  30. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  31. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  32. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  33. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  34. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  35. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  36. 2023KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigationcited by 38
  37. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  38. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35

50 most cited of 268 All 268 (JSON)