# Fine-Grained Alignment in Vision-and-Language Navigation through Bayesian Optimization

arXiv: https://arxiv.org/abs/2411.14811 · HTML: https://mlsanity.com/paper/2411.14811/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 46 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout](https://mlsanity.com/paper/1904.04195/) (2019)
- [Self-Monitoring Navigation Agent via Auxiliary Progress Estimation](https://mlsanity.com/paper/1901.03035/) (2019)
- [Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performance](https://mlsanity.com/paper/2304.11127/) (2023)
- [Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration](https://mlsanity.com/paper/2203.04006/) (2022)
- [Sparse Adversarial Video Attacks with Spatial Transformations](https://mlsanity.com/paper/2111.05468/) (2021)
- [Multimodal Contrastive Learning with Hard Negative Sampling for Human Activity Recognition](https://mlsanity.com/paper/2309.01262/) (2023)

## Built on by (most cited first, 2 of 2)

- [From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation](https://mlsanity.com/paper/2608.03143/) (2026)
- [Dynamic Context Adapters: Efficiently Infusing History into Vision-and-Language Models](https://mlsanity.com/paper/2608.10525/) (2026)

## Abstract (arXiv)

This paper addresses the challenge of fine-grained alignment in Vision-and-Language Navigation (VLN) tasks, where robots navigate realistic 3D environments based on natural language instructions. Current approaches use contrastive learning to align language with visual trajectory sequences. Nevertheless, they encounter difficulties with fine-grained vision negatives. To enhance cross-modal embeddings, we introduce a novel Bayesian Optimization-based adversarial optimization framework for creating fine-grained contrastive vision samples. To validate the proposed methodology, we conduct a series of experiments to assess the effectiveness of the enriched embeddings on fine-grained vision negatives. We conduct experiments on two common VLN benchmarks R2R and REVERIE, experiments on the them demonstrate that these embeddings benefit navigation, and can lead to a promising performance enhancement. Our source code and trained models are available at: https://anonymous.4open.science/r/FGVLN.