# FOAM: A Follower-aware Speaker Model For Vision-and-Language Navigation

arXiv: https://arxiv.org/abs/2206.04294 · HTML: https://mlsanity.com/paper/2206.04294/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [Meta Pseudo Labels](https://mlsanity.com/paper/2003.10580/) (2020)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation](https://mlsanity.com/paper/1811.10092/) (2018)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training](https://mlsanity.com/paper/2002.10638/) (2020)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks](https://mlsanity.com/paper/1911.07883/) (2019)
- [Improving Vision-and-Language Navigation with Image-Text Pairs from the Web](https://mlsanity.com/paper/2004.14973/) (2020)
- [Self-Monitoring Navigation Agent via Auxiliary Progress Estimation](https://mlsanity.com/paper/1901.03035/) (2019)
- [General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping](https://mlsanity.com/paper/1907.05446/) (2019)
- [Language and Visual Entity Relationship Graph for Agent Navigation](https://mlsanity.com/paper/2010.09304/) (2020)
- [Structured Scene Memory for Vision-Language Navigation](https://mlsanity.com/paper/2103.03454/) (2021)
- [Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation](https://mlsanity.com/paper/1803.07729/) (2018)
- [Robust Navigation with Language Pretraining and Stochastic Sampling](https://mlsanity.com/paper/1909.02244/) (2019)
- [Vision-Language Navigation with Random Environmental Mixup](https://mlsanity.com/paper/2106.07876/) (2021)
- [Unified Pragmatic Models for Generating and Following Instructions](https://mlsanity.com/paper/1711.04987/) (2017)
- [BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps](https://mlsanity.com/paper/2005.04625/) (2020)

## Built on by (most cited first, 22 of 22)

- [GridMM: Grid Memory Map for Vision-and-Language Navigation](https://mlsanity.com/paper/2307.12907/) (2023)
- [Scaling Data Generation in Vision-and-Language Navigation](https://mlsanity.com/paper/2307.15644/) (2023)
- [NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models](https://mlsanity.com/paper/2407.12366/) (2024)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024)
- [Lana: A Language-Capable Navigator for Instruction Following and Generation](https://mlsanity.com/paper/2303.08409/) (2023)
- [VLN-Trans: Translator for the Vision and Language Navigation Agent](https://mlsanity.com/paper/2302.09230/) (2023)
- [Controllable Navigation Instruction Generation with Chain of Thought Prompting](https://mlsanity.com/paper/2407.07433/) (2024)
- [Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheel](https://mlsanity.com/paper/2412.08467/) (2024)
- [FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Tasks](https://mlsanity.com/paper/2503.13966/) (2025)
- [SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts](https://mlsanity.com/paper/2412.05552/) (2024)
- [VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigation](https://mlsanity.com/paper/2402.03561/) (2024)
- [Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation](https://mlsanity.com/paper/2307.13368/) (2023)
- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigation](https://mlsanity.com/paper/2305.11918/) (2023)
- [MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigation](https://mlsanity.com/paper/2406.17960/) (2024)
- [Masked Path Modeling for Vision-and-Language Navigation](https://mlsanity.com/paper/2305.14268/) (2023)
- [NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired People](https://mlsanity.com/paper/2410.18109/) (2024)
- [A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigation](https://mlsanity.com/paper/2607.09792/) (2026)
- [Causality-based Cross-Modal Representation Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2403.03405/) (2024)
- [Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videos](https://mlsanity.com/paper/2602.23937/) (2026)
- [Pragmatics Meets Culture: Culturally-adapted Artwork Description Generation and Evaluation](https://mlsanity.com/paper/2604.02557/) (2026)

## Abstract (arXiv)

The speaker-follower models have proven to be effective in vision-and-language navigation, where a speaker model is used to synthesize new instructions to augment the training data for a follower navigation model. However, in many of the previous methods, the generated instructions are not directly trained to optimize the performance of the follower. In this paper, we present textscfoam, a textscFollower-textscaware speaker textscModel that is constantly updated given the follower feedback, so that the generated instructions can be more suitable to the current learning state of the follower. Specifically, we optimize the speaker using a bi-level optimization framework and obtain its training signals by evaluating the follower on labeled data. Experimental results on the Room-to-Room and Room-across-Room datasets demonstrate that our methods can outperform strong baseline models across settings. Analyses also reveal that our generated instructions are of higher quality than the baselines.