mlsanityData to 7 Oct 2026

Meta-Explore: Exploratory Hierarchical Vision-and-Language Navigation Using Scene Object Spectrum Grounding

arXiv 2303.04077 ↗First posted 7 Mar 2023cs.CV cs.AI cs.ROcited by 18

Not on the map. Why?

Abstract (arXiv)

The main challenge in vision-and-language navigation (VLN) is how to understand natural-language instructions in an unseen environment. The main limitation of conventional VLN algorithms is that if an action is mistaken, the agent fails to follow the instructions or explores unnecessary regions, leading the agent to an irrecoverable path. To tackle this problem, we propose Meta-Explore, a hierarchical navigation method deploying an exploitation policy to correct misled recent actions. We show that an exploitation policy, which moves the agent toward a well-chosen local goal among unvisited but observable states, outperforms a method which moves the agent to a previously visited state. We also highlight the demand for imagining regretful explorations with semantically meaningful clues. The key to our approach is understanding the object placements around the agent in spectral-domain. Specifically, we present a novel visual representation, called scene object spectrum (SOS), which performs category-wise 2D Fourier transform of detected objects. Combining exploitation policy and SOS features, the agent can correct its path by choosing a promising local goal. We evaluate our method in three VLN benchmarks: R2R, SOON, and REVERIE. Meta-Explore outperforms other baselines and shows significant generalization performance. In addition, local goal search using the proposed spectral-domain SOS features significantly improves the success rate by 17.1% and SPL by 20.6% for the SOON benchmark.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 53 of its 71 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

53 of 71 references.

References · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  11. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
  12. 2018On Evaluation of Embodied Navigation Agentscited by 526
38 more
  1. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  2. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  3. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  4. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  5. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  6. 2017Embodied Question Answeringcited by 335
  7. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  8. 2020Learning to Explore using Active Neural SLAMcited by 308
  9. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  10. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  11. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  12. 2020First return, then explorecited by 256
  13. 2019Vision-and-Dialog Navigationcited by 246
  14. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  15. 2020Neural Topological SLAM for Visual Navigationcited by 193
  16. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  17. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  18. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  19. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  20. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  21. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  22. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  23. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  24. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  25. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  26. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  27. 2019Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigationcited by 81
  28. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  29. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  30. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  31. 2019Executing Instructions in Situated Collaborative Interactionscited by 68
  32. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 63
  33. 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
  34. 2021Scene-Intuitive Agent for Remote Embodied Visual Groundingcited by 49
  35. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  36. 2021Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigationcited by 47
  37. 2019The StreetLearn Environment and Datasetcited by 42
  38. 2019Learning To Follow Directions in Street Viewcited by 29

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 6; 2026: 2.

18 papers.

Sort:

By area:

  1. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  2. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  3. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  4. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  5. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  6. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  7. 2024OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representationcited by 17
  8. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 16
  9. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  10. 2024Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigationcited by 15
  11. 2024PRET: Planning with Directed Fidelity Trajectory for Vision and Language Navigationcited by 6
  12. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
6 more
  1. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  2. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  3. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  4. 2024Open-Set 3D Semantic Instance Maps for Vision Language Navigation -- O3D-SIMcited by 2
  5. 2025Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigationcited by 1
  6. 2026Implicit Geometry Representations for Vision-and-Language Navigation from Web Videoscited by 0