mlsanityData to 7 Oct 2026

FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation

arXiv 2206.04294 ↗First posted 9 Jun 2022cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

The speaker-follower models have proven to be effective in vision-and-language navigation, where a speaker model is used to synthesize new instructions to augment the training data for a follower navigation model. However, in many of the previous methods, the generated instructions are not directly trained to optimize the performance of the follower. In this paper, we present textscfoam, a textscFollower-textscaware speaker textscModel that is constantly updated given the follower feedback, so that the generated instructions can be more suitable to the current learning state of the follower. Specifically, we optimize the speaker using a bi-level optimization framework and obtain its training signals by evaluating the follower on labeled data. Experimental results on the Room-to-Room and Room-across-Room datasets demonstrate that our methods can outperform strong baseline models across settings. Analyses also reveal that our generated instructions are of higher quality than the baselines.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

35 of 39 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2018On Evaluation of Embodied Navigation Agentscited by 526
  10. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  11. 2020Meta Pseudo Labelscited by 372
  12. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
23 more
  1. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  2. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  3. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  4. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  5. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  6. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  7. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  8. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  9. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  10. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  11. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  12. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  13. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  14. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  15. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  16. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  17. 2017Unified Pragmatic Models for Generating and Following Instructionscited by 55
  18. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  19. 2019Optimizing Data Usage via Differentiable Rewardscited by 48
  20. 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45
  21. 2019compare-mt: A Tool for Holistic Comparison of Language Generation Systemscited by 43
  22. 2020Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rulecited by 22
  23. 2021Meta Back-translationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 10; 2025: 2; 2026: 3.

22 papers.

Sort:

By area:

  1. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  2. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  3. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  4. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  5. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  6. 2023Lana: A Language-Capable Navigator for Instruction Following and Generationcited by 39
  7. 2023VLN-Trans: Translator for the Vision and Language Navigation Agentcited by 25
  8. 2024Controllable Navigation Instruction Generation with Chain of Thought Promptingcited by 20
  9. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  10. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  11. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  12. 2024VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigationcited by 14
10 more
  1. 2023Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generationcited by 11
  2. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  3. 2023PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigationcited by 8
  4. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  5. 2023Masked Path Modeling for Vision-and-Language Navigationcited by 4
  6. 2024NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired Peoplecited by 2
  7. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1
  8. 2024Causality-based Cross-Modal Representation Learning for Vision-and-Language Navigationcited by 0
  9. 2026Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videoscited by 0
  10. 2026Pragmatics Meets Culture: Culturally-adapted Artwork Description Generation and Evaluationcited by 0