# SOS-Match: Segmentation for Open-Set Robust Correspondence Search and Robot Localization in Unstructured Environments

arXiv: https://arxiv.org/abs/2401.04791 · HTML: https://mlsanity.com/paper/2401.04791/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 9 Jan 2024.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [D2-Net: A Trainable CNN for Joint Detection and Description of Local Features](https://mlsanity.com/paper/1905.03561/) (2019)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [Rethinking Visual Geo-localization for Large-Scale Applications](https://mlsanity.com/paper/2204.02287/) (2022)
- [AnyLoc: Towards Universal Visual Place Recognition](https://mlsanity.com/paper/2308.00688/) (2023)
- [MixVPR: Feature Mixing for Visual Place Recognition](https://mlsanity.com/paper/2303.02190/) (2023)
- [Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems](https://mlsanity.com/paper/2106.14386/) (2021)
- [CLIPPER: A Graph-Theoretic Framework for Robust Data Association](https://mlsanity.com/paper/2011.10202/) (2020)
- [Global Localization in Unstructured Environments using Semantic Object Maps Built from Various Viewpoints](https://mlsanity.com/paper/2303.04658/) (2023)
- [LSVL: Large-scale season-invariant visual localization for UAVs](https://mlsanity.com/paper/2212.03581/) (2022)
- [PUMA: Fully Decentralized Uncertainty-aware Multiagent Trajectory Planner with Real-time Image Segmentation-based Frame Alignment](https://mlsanity.com/paper/2311.03655/) (2023)

## Built on by (most cited first, 3 of 3)

- [Open-Set Semantic Uncertainty Aware Metric-Semantic Graph Matching](https://mlsanity.com/paper/2409.11555/) (2024)
- [LunarLoc: Segment-Based Global Localization on the Moon](https://mlsanity.com/paper/2506.16940/) (2025)
- [VISTA: Monocular Segmentation-Based Mapping for Appearance and View-Invariant Global Localization](https://mlsanity.com/paper/2507.11653/) (2025)

## Abstract (arXiv)

We present SOS-Match, a novel framework for detecting and matching objects in unstructured environments. Our system consists of 1) a front-end mapping pipeline using a zero-shot segmentation model to extract object masks from images and track them across frames and 2) a frame alignment pipeline that uses the geometric consistency of object relationships to efficiently localize across a variety of conditions. We evaluate SOS-Match on the Batvik seasonal dataset which includes drone flights collected over a coastal plot of southern Finland during different seasons and lighting conditions. Results show that our approach is more robust to changes in lighting and appearance than classical image feature-based approaches or global descriptor methods, and it provides more viewpoint invariance than learning-based feature detection and description approaches. SOS-Match localizes within a reference map up to 46x faster than other feature-based approaches and has a map size less than 0.5% the size of the most compact other maps. SOS-Match is a promising new approach for landmark detection and correspondence search in unstructured environments that is robust to changes in lighting and appearance and is more computationally efficient than other approaches, suggesting that the geometric arrangement of segments is a valuable localization cue in unstructured environments. We release our datasets at https://acl.mit.edu/SOS-Match/.