# The Wallpaper is Ugly: Indoor Localization using Vision and Language

arXiv: https://arxiv.org/abs/2410.03900 · HTML: https://mlsanity.com/paper/2410.03900/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [A Recurrent Vision-and-Language BERT for Navigation](https://mlsanity.com/paper/2011.13922/) (2020)
- [History Aware Multimodal Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2110.13309/) (2021)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Simple but Effective: CLIP Embeddings for Embodied AI](https://mlsanity.com/paper/2111.09888/) (2021)
- [A Survey of Vision-Language Pre-Trained Models](https://mlsanity.com/paper/2202.10936/) (2022)
- [SOON: Scenario Oriented Object Navigation with Graph-based Exploration](https://mlsanity.com/paper/2103.17138/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [Less is More: Generating Grounded Navigation Instructions from Landmarks](https://mlsanity.com/paper/2111.12872/) (2021)
- [Iterative Vision-and-Language Navigation](https://mlsanity.com/paper/2210.03087/) (2022)
- [Where Are You? Localization from Embodied Dialog](https://mlsanity.com/paper/2011.08277/) (2020)

## Built on by (most cited first, 1 of 1)

- [DialNav: Multi-turn Dialog Navigation with a Remote Guide](https://mlsanity.com/paper/2509.12894/) (2025)

## Abstract (arXiv)

We study the task of locating a user in a mapped indoor environment using natural language queries and images from the environment. Building on recent pretrained vision-language models, we learn a similarity score between text descriptions and images of locations in the environment. This score allows us to identify locations that best match the language query, estimating the user's location. Our approach is capable of localizing on environments, text, and images that were not seen during training. One model, finetuned CLIP, outperformed humans in our evaluation.