# Behavioral Analysis of Vision-and-Language Navigation Agents

arXiv: https://arxiv.org/abs/2307.10790 · HTML: https://mlsanity.com/paper/2307.10790/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 20 Jul 2023.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [A Recurrent Vision-and-Language BERT for Navigation](https://mlsanity.com/paper/2011.13922/) (2020)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Airbert: In-domain Pretraining for Vision-and-Language Navigation](https://mlsanity.com/paper/2108.09105/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping](https://mlsanity.com/paper/1907.05446/) (2019)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning](https://mlsanity.com/paper/2210.03112/) (2022)
- [SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.14143/) (2021)
- [Diagnosing Vision-and-Language Navigation: What Really Matters](https://mlsanity.com/paper/2103.16561/) (2021)

## Built on by (most cited first, 8 of 8)

- [Fast-Slow Test-Time Adaptation for Online Vision-and-Language Navigation](https://mlsanity.com/paper/2311.13209/) (2023)
- [Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigation](https://mlsanity.com/paper/2402.17587/) (2024)
- [Do Visual Imaginations Improve Vision-and-Language Navigation Agents?](https://mlsanity.com/paper/2503.16394/) (2025)
- [Mind the Error! Detection and Localization of Instruction Errors in Vision-and-Language Navigation](https://mlsanity.com/paper/2403.10700/) (2024)
- [DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2404.01994/) (2024)
- [Tell Me Where You Are: Multimodal LLMs Meet Place Recognition](https://mlsanity.com/paper/2406.17520/) (2024)
- [Hijacking Vision-and-Language Navigation Agents with Adversarial Environmental Attacks](https://mlsanity.com/paper/2412.02795/) (2024)
- [IntentionNav: A Benchmark for Intent-Driven Object Navigation from Implicit Human Instruction](https://mlsanity.com/paper/2605.23187/) (2026)

## Abstract (arXiv)

To be successful, Vision-and-Language Navigation (VLN) agents must be able to ground instructions to actions based on their surroundings. In this work, we develop a methodology to study agent behavior on a skill-specific basis -- examining how well existing agents ground instructions about stopping, turning, and moving towards specified objects or rooms. Our approach is based on generating skill-specific interventions and measuring changes in agent predictions. We present a detailed case study analyzing the behavior of a recent agent and then compare multiple agents in terms of skill-specific competency scores. This analysis suggests that biases from training have lasting effects on agent behavior and that existing models are able to ground simple referring expressions. Our comparisons between models show that skill-specific scores correlate with improvements in overall VLN task performance.