mlsanityData to 7 Oct 2026

DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learning

arXiv 2404.01994 ↗First posted 2 Apr 2024cs.CV cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language navigation (VLN) requires an agent to navigate in unseen environment by following natural language instruction. For task completion, the agent needs to align and integrate various navigation modalities, including instruction, observation and navigation history. Existing works primarily concentrate on cross-modal attention at the fusion stage to achieve this objective. Nevertheless, modality features generated by disparate uni-encoders reside in their own spaces, leading to a decline in the quality of cross-modal fusion and decision. To address this problem, we propose a Dual-levEL AligNment (DELAN) framework by cross-modal contrastive learning. This framework is designed to align various navigation-related modalities before fusion, thereby enhancing cross-modal interaction and action decision-making. Specifically, we divide the pre-fusion alignment into dual levels: instruction-history level and landmark-observation level according to their semantic correlations. We also reconstruct a dual-level instruction for adaptation to the dual-level alignment. As the training signals for pre-fusion alignment are extremely limited, self-supervised contrastive learning strategies are employed to enforce the matching between different modalities. Our approach seamlessly integrates with the majority of existing models, resulting in improved navigation performance on various VLN benchmarks, including R2R, R4R, RxR and CVDN.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 43 of its 47 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

43 of 47 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
31 more
  1. 2018On Evaluation of Embodied Navigation Agentscited by 526
  2. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  3. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  4. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  5. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  6. 2022X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrievalcited by 279
  7. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  8. 2019Vision-and-Dialog Navigationcited by 246
  9. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  10. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  11. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  12. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  13. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  14. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  15. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  16. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  17. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  18. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  19. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  20. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  21. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  22. 2020Sub-Instruction Aware Vision-and-Language Navigationcited by 64
  23. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  24. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  25. 2021Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigationcited by 20
  26. 2021Curriculum Learning for Vision-and-Language Navigationcited by 18
  27. 2021Contrastive Instruction-Trajectory Learning for Vision-Language Navigationcited by 16
  28. 2021VISITRON: Visual Semantics-Aligned Interactively Trained Object-Navigatorcited by 13
  29. 2022MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learningcited by 13
  30. 2023Behavioral Analysis of Vision-and-Language Navigation Agentscited by 8
  31. 2022LOViS: Learning Orientation and Visual Signals for Vision and Language Navigationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 4.

7 papers.

  1. 2024EmbSpatial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Modelscited by 100
  2. 2026SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigationcited by 8
  3. 2024To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigationcited by 1
  4. 2025Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservationcited by 1
  5. 2026CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learningcited by 1
  6. 2026Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videoscited by 0
  7. 2026Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigationcited by 0