mlsanityData to 14 Oct 2026

VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigator

arXiv 2105.11589 ↗First posted 25 May 2021cs.CV cs.AI cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

Interactive robots navigating photo-realistic environments need to be trained to effectively leverage and handle the dynamic nature of dialogue in addition to the challenges underlying vision-and-language navigation (VLN). In this paper, we present VISITRON, a multi-modal Transformer-based navigator better suited to the interactive regime inherent to Cooperative Vision-and-Dialog Navigation (CVDN). VISITRON is trained to: i) identify and associate object-level concepts and semantics between the environment and dialogue history, ii) identify when to interact vs. navigate via imitation learning of a binary classification head. We perform extensive pre-training and fine-tuning ablations with VISITRON to gain empirical insights and improve performance on CVDN. VISITRON's ability to identify when to interact leads to a natural generalization of the game-play mode introduced by Roman et al. (arXiv:2005.00728) for enabling the use of such models in different environments. VISITRON is competitive with models on the static CVDN leaderboard and attains state-of-the-art performance on the Success weighted by Path Length (SPL) metric.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 35 of its 43 references that are in the map. First posted on arXiv 25 May 2021.

Builds on

35 of 43 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  10. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  11. 2018On Evaluation of Embodied Navigation Agentscited by 547
  12. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
23 more
  1. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  2. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  3. 2019Vision-and-Dialog Navigationcited by 250
  4. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  5. 2020Experience Grounds Languagecited by 242
  6. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  7. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  8. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  9. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  10. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 130
  11. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 105
  12. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 89
  13. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  14. 2020Beyond Domain APIs: Task-oriented Conversational Modeling with Unstructured Knowledge Accesscited by 58
  15. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  16. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  17. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  18. 2019Chasing Ghosts: Instruction Following as Bayesian State Trackingcited by 41
  19. 2020Diagnosing the Environment Bias in Vision-and-Language Navigationcited by 41
  20. 2019EvalAI: Towards Better Evaluation Systems for AI Agentscited by 34
  21. 2020RMM: A Recursive Mental Model for Dialog Navigationcited by 32
  22. 2020Vision-Dialog Navigation by Exploring Cross-modal Memorycited by 23
  23. 2020Are Neural Open-Domain Dialog Systems Robust to Speech Recognition Errors in the Dialog History? An Empirical Studycited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 2; 2023: 3; 2024: 3; 2025: 0; 2026: 1.

13 papers.

Sort:

  1. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  2. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  3. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 89
  4. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 58
  5. 2023Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigationcited by 27
  6. 2022Dialog Acts for Task-Driven Embodied Agentscited by 13
  7. 2023ESceme: Vision-and-Language Navigation with Episodic Scene Memorycited by 10
  8. 2024DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learningcited by 7
  9. 2022DoRO: Disambiguation of referred object for embodied agentscited by 5
  10. 2023ELBA: Learning by Asking for Embodied Visual Navigation and Task Completioncited by 3
  11. 2024Multi-Task Domain Adaptation for Language Grounding with 3D Objectscited by 2
  12. 2024I2EDL: Interactive Instruction Error Detection and Localizationcited by 1
1 more
  1. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
  2. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
  3. 2026Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigation
  4. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  5. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigation
  6. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
  7. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agents
  8. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs