mlsanityData to 14 Oct 2026

The Wallpaper is Ugly: Indoor Localization using Vision and Language

arXiv 2410.03900 ↗First posted 4 Oct 2024cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

We study the task of locating a user in a mapped indoor environment using natural language queries and images from the environment. Building on recent pretrained vision-language models, we learn a similarity score between text descriptions and images of locations in the environment. This score allows us to identify locations that best match the language query, estimating the user's location. Our approach is capable of localizing on environments, text, and images that were not seen during training. One model, finetuned CLIP, outperformed humans in our evaluation.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 27 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

22 of 27 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  8. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  9. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  10. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  11. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  12. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
10 more
  1. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  2. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  3. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 239
  4. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  5. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  6. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  7. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  8. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 64
  9. 2022Iterative Vision-and-Language Navigationcited by 35
  10. 2020Where Are You? Localization from Embodied Dialogcited by 24

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025DialNav: Multi-turn Dialog Navigation with a Remote Guidecited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Where Am I? Semantic Map Grounding via Vision-Language Models for Multi-Modal Localization
  2. 2024Teaching VLMs to Localize Specific Objects from In-context Examples
  3. 2024LocateBench: Evaluating the Locating Ability of Vision Language Models
  4. 2025Fine-Tuning Vision-Language Models for Visual Navigation Assistance
  5. 2026VLM-GLoc: Vision-Language Model Enhanced Monte Carlo Localization for Robust Semantic Global Localization in Cluttered Quasi-Static Environments
  6. 2026What Words Keep of a Place: Zero-Shot Language Reasoning for Cross-View Geo-Localization
  7. 2025Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales
  8. 2026LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation