mlsanityData to 14 Oct 2026

Structure-Encoding Auxiliary Tasks for Improved Visual Representation in Vision-and-Language Navigation

arXiv 2211.11116 ↗First posted 20 Nov 2022cs.CV cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

In Vision-and-Language Navigation (VLN), researchers typically take an image encoder pre-trained on ImageNet without fine-tuning on the environments that the agent will be trained or tested on. However, the distribution shift between the training images from ImageNet and the views in the navigation environments may render the ImageNet pre-trained image encoder suboptimal. Therefore, in this paper, we design a set of structure-encoding auxiliary tasks (SEA) that leverage the data in the navigation environments to pre-train and improve the image encoder. Specifically, we design and customize (1) 3D jigsaw, (2) traversability prediction, and (3) instance classification to pre-train the image encoder. Through rigorous ablations, our SEA pre-trained features are shown to better encode structural information of the scenes, which ImageNet pre-trained features fail to properly encode but is crucial for the target navigation task. The SEA pre-trained features can be easily plugged into existing VLN agents without any tuning. For example, on Test-Unseen environments, the VLN agents combined with our SEA pre-trained features achieve absolute success rate improvement of 12% for Speaker-Follower, 5% for Env-Dropout, and 4% for AuxRN.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 54 of its 57 references that are in the map. First posted on arXiv 20 Nov 2022.

Builds on

54 of 57 references.

References · 54

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  8. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  9. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  11. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  12. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
38 more
  1. 2019Contrastive Multiview Codingcited by 1,301
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  4. 2016Colorful Image Colorizationcited by 1,114
  5. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  8. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  10. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  11. 2018On Evaluation of Embodied Navigation Agentscited by 547
  12. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  13. 2020Rethinking Pre-training and Self-trainingcited by 352
  14. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  15. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  16. 2016Learning Representations for Automatic Colorizationcited by 251
  17. 2019Vision-and-Dialog Navigationcited by 250
  18. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  19. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  20. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  21. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  22. 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
  23. 2020Neural Topological SLAM for Visual Navigationcited by 195
  24. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  25. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  26. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  27. 2017Colorization as a Proxy Task for Visual Understandingcited by 175
  28. 2019Unsupervised State Representation Learning in Ataricited by 144
  29. 2016Learning Features by Watching Objects Movecited by 138
  30. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 130
  31. 2020What makes instance discrimination good for transfer learning?cited by 111
  32. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 105
  33. 2015Learning to See by Movingcited by 102
  34. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  35. 2020Predictive Information Accelerates Learning in RLcited by 56
  36. 2020Auxiliary Tasks Speed Up Learning PointGoal Navigationcited by 54
  37. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 48
  38. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 2; 2026: 1.

9 papers.

  1. 2023Learning Vision-and-Language Navigation from YouTube Videoscited by 47
  2. 2023Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigationcited by 18
  3. 2024Vision-Language Navigation with Embodied Intelligence: A Surveycited by 9
  4. 2025VISTAv2: World Imagination for Indoor Vision-and-Language Navigationcited by 6
  5. 2024Hierarchical Spatial Proximity Reasoning for Vision-and-Language Navigationcited by 2
  6. 2024Reinforcement Learning via Auxiliary Task Distillationcited by 2
  7. 2024NaVIP: An Image-Centric Indoor Navigation Solution for Visually Impaired Peoplecited by 2
  8. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1
  9. 2025Adept: Annotation-Denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretrainingcited by 0