# CLIP-Loc: Multi-modal Landmark Association for Global Localization in Object-based Maps

arXiv: https://arxiv.org/abs/2402.06092 · HTML: https://mlsanity.com/paper/2402.06092/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://mlsanity.com/paper/2203.10421/) (2022)
- [MAGSAC++, a fast, reliable and accurate robust estimator](https://mlsanity.com/paper/1912.05909/) (2019)
- [CubeSLAM: Monocular 3D Object SLAM](https://mlsanity.com/paper/1806.00557/) (2018)
- [QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAM](https://mlsanity.com/paper/1804.04011/) (2018)
- [Monocular Object and Plane SLAM in Structured Environments](https://mlsanity.com/paper/1809.03415/) (2018)
- [SO-SLAM: Semantic Object SLAM with Scale Proportional and Symmetrical Texture Constraints](https://mlsanity.com/paper/2109.04884/) (2021)
- [OA-SLAM: Leveraging Objects for Camera Relocalization in Visual SLAM](https://mlsanity.com/paper/2209.08338/) (2022)
- [An Object SLAM Framework for Association, Mapping, and High-Level Tasks](https://mlsanity.com/paper/2305.07299/) (2023)
- [FM-Loc: Using Foundation Models for Improved Vision-based Localization](https://mlsanity.com/paper/2304.07058/) (2023)
- [SQ-SLAM: Monocular Semantic SLAM Based on Superquadric Object Representation](https://mlsanity.com/paper/2209.10817/) (2022)

## Built on by (most cited first, 8 of 8)

- ["Where am I?" Scene Retrieval with Language](https://mlsanity.com/paper/2404.14565/) (2024)
- [SegMASt3R: Geometry Grounded Segment Matching](https://mlsanity.com/paper/2510.05051/) (2025)
- [CLIP-Clique: Graph-based Correspondence Matching Augmented by Vision Language Models for Object-based Global Localization](https://mlsanity.com/paper/2410.03054/) (2024)
- [Semantic Enhancement for Object SLAM with Heterogeneous Multimodal Large Language Model Agents](https://mlsanity.com/paper/2411.06752/) (2024)
- [MVL-Loc: Leveraging Vision-Language Model for Generalizable Multi-Scene Camera Relocalization](https://mlsanity.com/paper/2507.04509/) (2025)
- [MSG-Loc: Multi-Label Likelihood-based Semantic Graph Matching for Object-Level Global Localization](https://mlsanity.com/paper/2512.03522/) (2025)
- [Compact Object-Level Representations with Open-Vocabulary Understanding for Indoor Visual Relocalization](https://mlsanity.com/paper/2606.24767/) (2026)
- [Towards Global Localization using Multi-Modal Object-Instance Re-Identification](https://mlsanity.com/paper/2409.12002/) (2024)

## Abstract (arXiv)

This paper describes a multi-modal data association method for global localization using object-based maps and camera images. In global localization, or relocalization, using object-based maps, existing methods typically resort to matching all possible combinations of detected objects and landmarks with the same object category, followed by inlier extraction using RANSAC or brute-force search. This approach becomes infeasible as the number of landmarks increases due to the exponential growth of correspondence candidates. In this paper, we propose labeling landmarks with natural language descriptions and extracting correspondences based on conceptual similarity with image observations using a Vision Language Model (VLM). By leveraging detailed text information, our approach efficiently extracts correspondences compared to methods using only object categories. Through experiments, we demonstrate that the proposed method enables more accurate global localization with fewer iterations compared to baseline methods, exhibiting its efficiency.