mlsanityData to 14 Oct 2026

CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations

arXiv 2207.02185 ↗First posted 5 Jul 2022cs.CV cs.AI cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation (VLN) tasks require an agent to navigate through the environment based on language instructions. In this paper, we aim to solve two key challenges in this task: utilizing multilingual instructions for improved instruction-path grounding and navigating through new environments that are unseen during training. To address these challenges, we propose CLEAR: Cross-Lingual and Environment-Agnostic Representations. First, our agent learns a shared and visually-aligned cross-lingual language representation for the three languages (English, Hindi and Telugu) in the Room-Across-Room dataset. Our language representation learning is guided by text pairs that are aligned by visual information. Second, our agent learns an environment-agnostic visual representation by maximizing the similarity between semantically-aligned image pairs (with constraints on object-matching) from different environments. Our environment agnostic visual representation can mitigate the environment bias induced by low-level visual information. Empirically, on the Room-Across-Room dataset, we show that our multilingual agent gets large improvements in all metrics over the strong baseline model when generalizing to unseen environments with the cross-lingual language representation and the environment-agnostic visual representation. Furthermore, we show that our learned language and visual representations can be successfully transferred to the Room-to-Room and Cooperative Vision-and-Dialogue Navigation task, and present detailed qualitative and quantitative generalization and grounding analysis. Our code is available at https://github.com/jialuli-luka/CLEAR

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 57 of its 72 references that are in the map. First posted on arXiv 5 Jul 2022.

Builds on

57 of 72 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2017Mask R-CNNcited by 7,277
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  11. 2018On Evaluation of Embodied Navigation Agentscited by 547
  12. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
38 more
  1. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  2. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  3. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  4. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  5. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  6. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  7. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  8. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  9. 2019Vision-and-Dialog Navigationcited by 250
  10. 2019Massively Multilingual Neural Machine Translationcited by 246
  11. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  12. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  13. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  14. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  15. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  16. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 130
  17. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  18. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  19. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 105
  20. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 105
  21. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  22. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  23. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  24. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  25. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  26. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  27. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  28. 2020Active Visual Information Gathering for Vision-Language Navigationcited by 55
  29. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  30. 2020BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Stepscited by 49
  31. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 48
  32. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  33. 2020Diagnosing the Environment Bias in Vision-and-Language Navigationcited by 41
  34. 2020Soft Expert Reward Learning for Vision-and-Language Navigationcited by 37
  35. 2017Fluency-Guided Cross-Lingual Image Captioningcited by 33
  36. 2021Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Informationcited by 31
  37. 2018Unpaired Image Captioning by Language Pivotingcited by 27
  38. 2020Visual Grounding in Video for Unsupervised Word Translationcited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 4; 2025: 1; 2026: 1.

10 papers.

  1. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 122
  2. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  3. 2023PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigationcited by 77
  4. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  5. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  6. 2024Why Only Text: Empowering Vision-and-Language Navigation with Multi-modal Promptscited by 9
  7. 2024MAGIC: Meta-Ability Guided Interactive Chain-of-Distillation for Effective-and-Efficient Vision-and-Language Navigationcited by 6
  8. 2025PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigationcited by 6
  9. 2024Into the Unknown: Generating Geospatial Descriptions for New Environmentscited by 1
  10. 2026Query, Align, and Distill: Navigation-Aware Cross-Modal Interaction for Efficient Vision-and-Language Navigationcited by 0