# Matching Anything by Segmenting Anything

arXiv: https://arxiv.org/abs/2406.04221 · HTML: https://mlsanity.com/paper/2406.04221/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 79 of its 82 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 30 of 79)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)

## Built on by (most cited first, 30 of 50)

- [DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks](https://mlsanity.com/paper/2502.17157/) (2025)
- [UFM: A Simple Path towards Unified Dense Correspondence with Flow](https://mlsanity.com/paper/2506.09278/) (2025)
- [Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking](https://mlsanity.com/paper/2410.01806/) (2024)
- [SAM2MOT: A Novel Paradigm of Multi-Object Tracking by Segmentation](https://mlsanity.com/paper/2504.04519/) (2025)
- [Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2501.04670/) (2025)
- [Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory](https://mlsanity.com/paper/2411.16934/) (2024)
- [OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2503.10616/) (2025)
- [Open-World Object Counting in Videos](https://mlsanity.com/paper/2506.15368/) (2025)
- [SLAck: Semantic, Location, and Appearance Aware Open-Vocabulary Tracking](https://mlsanity.com/paper/2409.11235/) (2024)
- [No Train Yet Gain: Towards Generic Multi-Object Tracking in Sports and Beyond](https://mlsanity.com/paper/2506.01373/) (2025)
- [3D-Aware Instance Segmentation and Tracking in Egocentric Videos](https://mlsanity.com/paper/2408.09860/) (2024)
- [Puzzles: Unbounded Video-Depth Augmentation for Scalable End-to-End 3D Reconstruction](https://mlsanity.com/paper/2506.23863/) (2025)
- [SegMASt3R: Geometry Grounded Segment Matching](https://mlsanity.com/paper/2510.05051/) (2025)
- [Walker: Self-supervised Multiple Object Tracking by Walking on Temporal Appearance Graphs](https://mlsanity.com/paper/2409.17221/) (2024)
- [Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs](https://mlsanity.com/paper/2411.06702/) (2024)
- [VideoSAM: Open-World Video Segmentation](https://mlsanity.com/paper/2410.08781/) (2024)
- [Slot-Level Robotic Placement via Visual Imitation from Single Human Video](https://mlsanity.com/paper/2504.01959/) (2025)
- [VGGT-Segmentor: Geometry-Enhanced Cross-View Segmentation](https://mlsanity.com/paper/2604.13596/) (2026)
- [History-Aware Transformation of ReID Features for Multiple Object Tracking](https://mlsanity.com/paper/2503.12562/) (2025)
- [CrowdTrack: A Benchmark for Difficult Multiple Pedestrian Tracking in Real Scenarios](https://mlsanity.com/paper/2507.02479/) (2025)
- [Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2510.10671/) (2025)
- [Neural USD: An object-centric framework for iterative editing and control](https://mlsanity.com/paper/2510.23956/) (2025)
- [Temporally Propagated Masks and Bounding Boxes: Combining the Best of Both Worlds for Multi-Object Tracking](https://mlsanity.com/paper/2409.14220/) (2024)
- [Attention to Trajectory: Trajectory-Aware Open-Vocabulary Tracking](https://mlsanity.com/paper/2503.08145/) (2025)
- [Visibility-Uncertainty-guided 3D Gaussian Inpainting via Scene Conceptional Learning](https://mlsanity.com/paper/2504.17815/) (2025)
- [ViRefSAM: Visual Reference-Guided Segment Anything Model for Remote Sensing Segmentation](https://mlsanity.com/paper/2507.02294/) (2025)
- [Tracking the Unstable: Appearance-Guided Motion Modeling for Robust Multi-Object Tracking in UAV-Captured Videos](https://mlsanity.com/paper/2508.01730/) (2025)
- [Rethinking Pose Refinement in 3D Gaussian Splatting under Pose Prior and Geometric Uncertainty](https://mlsanity.com/paper/2603.16538/) (2026)
- [SAMIDARE: Advanced Tracking-by-Segmentation for Dense Scenarios](https://mlsanity.com/paper/2604.22162/) (2026)
- [OW-Rep: Open World Object Detection with Instance Representation Learning](https://mlsanity.com/paper/2409.16073/) (2024)

## Abstract (arXiv)

The robust association of the same objects across video frames in complex scenes is crucial for many applications, especially Multiple Object Tracking (MOT). Current methods predominantly rely on labeled domain-specific video datasets, which limits the cross-domain generalization of learned similarity embeddings. We propose MASA, a novel method for robust instance association learning, capable of matching any objects within videos across diverse domains without tracking labels. Leveraging the rich object segmentation from the Segment Anything Model (SAM), MASA learns instance-level correspondence through exhaustive data transformations. We treat the SAM outputs as dense object region proposals and learn to match those regions from a vast image collection. We further design a universal MASA adapter which can work in tandem with foundational segmentation or detection models and enable them to track any detected objects. Those combinations present strong zero-shot tracking ability in complex domains. Extensive tests on multiple challenging MOT and MOTS benchmarks indicate that the proposed method, using only unlabeled static images, achieves even better performance than state-of-the-art methods trained with fully annotated in-domain video sequences, in zero-shot association. Project Page: https://matchinganything.github.io/