mlsanityData to 14 Oct 2026

Scene-Intuitive Agent for Remote Embodied Visual Grounding

arXiv 2103.12944 ↗First posted 24 Mar 2021cs.CVcited by 49

Not on the map. Why?

Abstract (arXiv)

Humans learn from life events to form intuitions towards the understanding of visual environments and languages. Envision that you are instructed by a high-level instruction, "Go to the bathroom in the master bedroom and replace the blue towel on the left wall", what would you possibly do to carry out the task? Intuitively, we comprehend the semantics of the instruction to form an overview of where a bathroom is and what a blue towel is in mind; then, we navigate to the target location by consistently matching the bathroom appearance in mind with the current scene. In this paper, we present an agent that mimics such human behaviors. Specifically, we focus on the Remote Embodied Visual Referring Expression in Real Indoor Environments task, called REVERIE, where an agent is asked to correctly localize a remote target object specified by a concise high-level natural language instruction, and propose a two-stage training pipeline. In the first stage, we pretrain the agent with two cross-modal alignment sub-tasks, namely the Scene Grounding task and the Object Grounding task. The agent learns where to stop in the Scene Grounding task and what to attend to in the Object Grounding task respectively. Then, to generate action sequences, we propose a memory-augmented attentive action decoder to smoothly fuse the pre-trained vision and language representations with the agent's past memory experiences. Without bells and whistles, experimental results show that our method outperforms previous state-of-the-art(SOTA) significantly, demonstrating the effectiveness of our method.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 38 of its 51 references that are in the map. First posted on arXiv 24 Mar 2021.

Builds on

38 of 51 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2017Mask R-CNNcited by 7,277
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
26 more
  1. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  2. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  3. 2017Embodied Question Answeringcited by 336
  4. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  5. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  6. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  7. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  8. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  9. 2017Cognitive Mapping and Planning for Visual Navigationcited by 212
  10. 2018Semi-parametric Topological Memory for Navigationcited by 209
  11. 2016Learning to Navigate in Complex Environmentscited by 198
  12. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  13. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  14. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  15. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  16. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 107
  17. 2017Neural Map: Structured Memory for Deep Reinforcement Learningcited by 90
  18. 2019Improving Referring Expression Grounding with Cross-modal Attention-guided Erasingcited by 88
  19. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  20. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  21. 2016Control of Memory, Active Perception, and Action in Minecraftcited by 61
  22. 2020Graph-Structured Referring Expression Reasoning in The Wildcited by 58
  23. 2020Do RNN and LSTM have Long Memory?cited by 47
  24. 2019Chasing Ghosts: Instruction Following as Bayesian State Trackingcited by 41
  25. 2019Relationship-Embedded Representation Learning for Grounding Referring Expressionscited by 25
  26. 2018Visual Question Reasoning on General Dependency Treecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 8; 2023: 10; 2024: 14; 2025: 9; 2026: 6.

49 papers.

Sort:

By area:

  1. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  2. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  3. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 122
  4. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  5. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 117
  6. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  7. 2023Bird's-Eye-View Scene Graph for Vision-Language Navigationcited by 72
  8. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  9. 2022Target-Driven Structured Transformer Planner for Vision-Language Navigationcited by 58
  10. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  11. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 50
  12. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 50
37 more
  1. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  2. 2021Vision-Language Navigation: A Survey and Taxonomycited by 34
  3. 2023Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigationcited by 25
  4. 2022Causal Reasoning Meets Visual Representation Learning: A Prospective Studycited by 24
  5. 2023GeoVLN: Learning Geometry-Enhanced Visual Representation with Slot Attention for Vision-and-Language Navigationcited by 24
  6. 2023Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Groundingcited by 18
  7. 2023A Dual Semantic-Aware Recurrent Global-Adaptive Network For Vision-and-Language Navigationcited by 18
  8. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 17
  9. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  10. 2025DOPE: Dual Object Perception-Enhancement Network for Vision-and-Language Navigationcited by 14
  11. 2023RREx-BoT: Remote Referring Expressions with a Bag of Trickscited by 12
  12. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  13. 2023Multiple Thinking Achieving Meta-Ability Decoupling for Object Navigationcited by 8
  14. 2024Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigationcited by 8
  15. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  16. 2022Embodied Referring Expression for Manipulation Question Answering in Interactive Environmentcited by 7
  17. 2024Augmented Commonsense Knowledge for Remote Object Groundingcited by 7
  18. 2025PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigationcited by 6
  19. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  20. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 5
  21. 2023Actional Atomic-Concept Learning for Demystifying Vision-Language Navigationcited by 4
  22. 2025Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigationcited by 4
  23. 2025MSNav: Zero-Shot Vision-and-Language Navigation with Dynamic Memory and LLM Spatial Reasoningcited by 4
  24. 20263D Gaussian Map with Open-Set Semantic Grouping for Vision-Language Navigationcited by 4
  25. 2022Layout-aware Dreamer for Embodied Referring Expression Groundingcited by 3
  26. 2024Hierarchical Spatial Proximity Reasoning for Vision-and-Language Navigationcited by 2
  27. 2024AIGeN: An Adversarial Approach for Instruction Generation in VLNcited by 2
  28. 2026Uncertainty-Aware Gaussian Map for Vision-Language Navigationcited by 1
  29. 2024Causality-based Cross-Modal Representation Learning for Vision-and-Language Navigationcited by 0
  30. 2024Temporal-Spatial Object Relations Modeling for Vision-and-Language Navigationcited by 0
  31. 2024Vision-and-Language Navigation Generative Pretrained Transformercited by 0
  32. 2025Learning to Reason and Navigate: Parameter Efficient Action Planning with Large Language Modelscited by 0
  33. 2025Landmark-Guided Knowledge for Vision-and-Language Navigationcited by 0
  34. 2025TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Makingcited by 0
  35. 2026Beyond Textual Knowledge-Leveraging Multimodal Knowledge Bases for Enhancing Vision-and-Language Navigationcited by 0
  36. 2026ViSMoE: Visual-Aware Sparse Mixture-of-Experts for Embodied Referring Expression Groundingcited by 0
  37. 2026Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigationcited by 0