# VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models arXiv: https://arxiv.org/abs/2603.09826 · HTML: https://mlsanity.com/paper/2603.09826/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 47 of its 64 references that are in the map. - First posted on arXiv 10 Mar 2026. ## Placed in - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) ## Builds on (most cited first, 30 of 47) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016) - [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023) - [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024) - [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023) - [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023) - [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025) - [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025) - [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021) - [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025) - [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015) - [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021) - [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019) - [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024) - [PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition](https://mlsanity.com/paper/1804.03492/) (2018) - [Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds](https://mlsanity.com/paper/2112.08879/) (2021) - [LanguageRefer: Spatial-Language Model for 3D Visual Grounding](https://mlsanity.com/paper/2107.03438/) (2021) - [OrienterNet: Visual Localization in 2D Public Maps with Neural Matching](https://mlsanity.com/paper/2304.02009/) (2023) - [Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud](https://mlsanity.com/paper/2103.16381/) (2021) - [OverlapNet: Loop Closing for LiDAR-based SLAM](https://mlsanity.com/paper/2105.11344/) (2021) - [OverlapTransformer: An Efficient and Rotation-Invariant Transformer Network for LiDAR-Based Place Recognition](https://mlsanity.com/paper/2203.03397/) (2022) - [Vision-and-Language Navigation via Causal Learning](https://mlsanity.com/paper/2404.10241/) (2024) - [SensatUrban: Learning Semantics from Urban-Scale Photogrammetric Point Clouds](https://mlsanity.com/paper/2201.04494/) (2022) - [Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching](https://mlsanity.com/paper/2311.12751/) (2023) - [Multi-scale Interaction for Real-time LiDAR Data Segmentation on an Embedded Platform](https://mlsanity.com/paper/2008.09162/) (2020) ## Built on by (most cited first, 4 of 4) - [Geoparsing: Diagram Parsing for Plane and Solid Geometry with a Unified Formal Language](https://mlsanity.com/paper/2604.11600/) (2026) - [TOL: Textual Localization with OpenStreetMap](https://mlsanity.com/paper/2604.01644/) (2026) - [VLM-GLoc: Vision-Language Model Enhanced Monte Carlo Localization for Robust Semantic Global Localization in Cluttered Quasi-Static Environments](https://mlsanity.com/paper/2605.30506/) (2026) - [Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration](https://mlsanity.com/paper/2605.31196/) (2026) ## Abstract (arXiv) Text-to-point-cloud (T2P) localization aims to infer precise spatial positions within 3D point cloud maps from natural language descriptions, reflecting how humans perceive and communicate spatial layouts through language. However, existing methods largely rely on shallow text-point cloud correspondence without effective spatial reasoning, limiting their accuracy in complex environments. To address this limitation, we propose VLM-Loc, a framework that leverages the spatial reasoning capability of large vision-language models (VLMs) for T2P localization. Specifically, we transform point clouds into bird's-eye-view (BEV) images and scene graphs that jointly encode geometric and semantic context, providing structured inputs for the VLM to learn cross-modal representations bridging linguistic and spatial semantics. On top of these representations, we introduce a partial node assignment mechanism that explicitly associates textual cues with scene graph nodes, enabling interpretable spatial reasoning for accurate localization. To facilitate systematic evaluation across diverse scenes, we present CityLoc, a benchmark built from multi-source point clouds for fine-grained T2P localization. Experiments on CityLoc demonstrate VLM-Loc achieves superior accuracy and robustness compared to state-of-the-art methods. Our code, model, and dataset are available at hrefhttps://github.com/MCG-NKU/nku-3d-visionrepository.