# Tell Me Where You Are: Multimodal LLMs Meet Place Recognition arXiv: https://arxiv.org/abs/2406.17520 ยท HTML: https://mlsanity.com/paper/2406.17520/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. - Builds on 37 of its 52 references that are in the map. - First posted on arXiv 25 Jun 2024. ## Placed in - LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/) - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) ## Builds on (most cited first, 30 of 37) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020) - [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019) - [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021) - [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022) - [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020) - [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022) - [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020) - [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023) - [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022) - [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015) - [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023) - [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024) - [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017) - [DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model](https://mlsanity.com/paper/2310.01412/) (2023) - [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022) - [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023) - [Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition](https://mlsanity.com/paper/2103.01486/) (2021) - [Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving](https://mlsanity.com/paper/2310.01957/) (2023) - [PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition](https://mlsanity.com/paper/1804.03492/) (2018) - [AnyLoc: Towards Universal Visual Place Recognition](https://mlsanity.com/paper/2308.00688/) (2023) - [MixVPR: Feature Mixing for Visual Place Recognition](https://mlsanity.com/paper/2303.02190/) (2023) - [Optimal Transport Aggregation for Visual Place Recognition](https://mlsanity.com/paper/2311.15937/) (2023) - [TransVPR: Transformer-based place recognition with multi-level attention aggregation](https://mlsanity.com/paper/2201.02001/) (2022) - [R2Former: Unified Retrieval and Reranking Transformer for Place Recognition](https://mlsanity.com/paper/2304.03410/) (2023) - [ADAPT: Action-aware Driving Caption Transformer](https://mlsanity.com/paper/2302.00673/) (2023) - [SOE-Net: A Self-Attention and Orientation Encoding Network for Point Cloud based Place Recognition](https://mlsanity.com/paper/2011.12430/) (2020) - [SeqNet: Learning Descriptors for Sequence-based Hierarchical Place Recognition](https://mlsanity.com/paper/2102.11603/) (2021) ## Built on by (most cited first, 7 of 7) - [SkyVLN: Vision-and-Language Navigation and NMPC Control for UAVs in Urban Environments](https://mlsanity.com/paper/2507.06564/) (2025) - [Multiview Scene Graph](https://mlsanity.com/paper/2410.11187/) (2024) - [VGGT-MPR: VGGT-Enhanced Multimodal Place Recognition in Autonomous Driving Environments](https://mlsanity.com/paper/2602.19735/) (2026) - [Scale, Don't Fine-tune: Guiding Multimodal LLMs for Efficient Visual Place Recognition at Test-Time](https://mlsanity.com/paper/2509.02129/) (2025) - [DialogueVPR: Towards Conversational Visual Place Recognition](https://mlsanity.com/paper/2607.14115/) (2026) - [Semantics in Robotics: Environmental Data Can't Yield Conventions of Human Behaviour](https://mlsanity.com/paper/2410.19308/) (2024) - [G2IA: Geometry-Guided Instance-Aware Retrieval and Refinement for Cross-Modal Place Recognition](https://mlsanity.com/paper/2606.15287/) (2026) ## Abstract (arXiv) Large language models (LLMs) exhibit a variety of promising capabilities in robotics, including long-horizon planning and commonsense reasoning. However, their performance in place recognition is still underexplored. In this work, we introduce multimodal LLMs (MLLMs) to visual place recognition (VPR), where a robot must localize itself using visual observations. Our key design is to use vision-based retrieval to propose several candidates and then leverage language-based reasoning to carefully inspect each candidate for a final decision. Specifically, we leverage the robust visual features produced by off-the-shelf vision foundation models (VFMs) to obtain several candidate locations. We then prompt an MLLM to describe the differences between the current observation and each candidate in a pairwise manner, and reason about the best candidate based on these descriptions. Our results on three datasets demonstrate that integrating the general-purpose visual features from VFMs with the reasoning capabilities of MLLMs already provides an effective place recognition solution, without any VPR-specific supervised training. We believe our work can inspire new possibilities for applying and designing foundation models, i.e., VFMs, LLMs, and MLLMs, to enhance the localization and navigation of mobile robots.