# Learning Co-segmentation by Segment Swapping for Retrieval and Discovery

arXiv: https://arxiv.org/abs/2110.15904 · HTML: https://mlsanity.com/paper/2110.15904/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 82 references that are in the map.
- First posted on arXiv 29 Oct 2021.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition](https://mlsanity.com/paper/2103.01486/) (2021)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Learning with Average Precision: Training Image Retrieval with a Listwise Loss](https://mlsanity.com/paper/1906.07589/) (2019)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences](https://mlsanity.com/paper/1912.05524/) (2019)
- [Convolutional neural network architecture for geometric matching](https://mlsanity.com/paper/1703.05593/) (2017)
- [Learning Accurate Dense Correspondences and When to Trust Them](https://mlsanity.com/paper/2101.01710/) (2021)
- [Self-supervising Fine-grained Region Similarities for Large-scale Image Localization](https://mlsanity.com/paper/2006.03926/) (2020)
- [Occlusion Aware Unsupervised Learning of Optical Flow](https://mlsanity.com/paper/1711.05890/) (2017)
- [Efficient Neighbourhood Consensus Networks via Submanifold Sparse Convolutions](https://mlsanity.com/paper/2004.10566/) (2020)
- [The Fast Bilateral Solver](https://mlsanity.com/paper/1511.03296/) (2015)
- [Unsupervised Object Discovery and Localization in the Wild: Part-based Matching with Bottom-up Region Proposals](https://mlsanity.com/paper/1501.06170/) (2015)
- [DGC-Net: Dense Geometric Correspondence Network](https://mlsanity.com/paper/1810.08393/) (2018)
- [Toward unsupervised, multi-object discovery in large-scale image collections](https://mlsanity.com/paper/2007.02662/) (2020)
- [RANSAC-Flow: generic two-stage image alignment](https://mlsanity.com/paper/2004.01526/) (2020)

## Built on by (most cited first, 15 of 15)

- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut](https://mlsanity.com/paper/2209.00383/) (2022)
- [DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery](https://mlsanity.com/paper/2303.09813/) (2023)
- [ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives](https://mlsanity.com/paper/2411.19083/) (2024)
- [Unsupervised and semi-supervised co-salient object detection via segmentation frequency statistics](https://mlsanity.com/paper/2311.06654/) (2023)
- [O-MaMa: Learning Object Mask Matching between Egocentric and Exocentric Views](https://mlsanity.com/paper/2506.06026/) (2025)
- [Diffusion Models as Data Mining Tools](https://mlsanity.com/paper/2408.02752/) (2024)
- [Unsupervised Object Discovery: A Comprehensive Survey and Unified Taxonomy](https://mlsanity.com/paper/2411.00868/) (2024)
- [Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction](https://mlsanity.com/paper/2602.18996/) (2026)
- [LCCo: Lending CLIP to Co-Segmentation](https://mlsanity.com/paper/2308.11506/) (2023)
- [Self-supervised co-salient object detection via feature correspondence at multiple scales](https://mlsanity.com/paper/2403.11107/) (2024)
- [Learning Accurate Segmentation Purely from Self-Supervision](https://mlsanity.com/paper/2602.23759/) (2026)
- [Fusion of Infrared and Visible Images based on Spatial-Channel Attentional Mechanism](https://mlsanity.com/paper/2308.13672/) (2023)
- [PANC: Prior-Aware Normalized Cut via Anchor-Augmented Token Graphs](https://mlsanity.com/paper/2602.06912/) (2026)

## Abstract (arXiv)

The goal of this work is to efficiently identify visually similar patterns in images, e.g. identifying an artwork detail copied between an engraving and an oil painting, or recognizing parts of a night-time photograph visible in its daytime counterpart. Lack of training data is a key challenge for this co-segmentation task. We present a simple yet surprisingly effective approach to overcome this difficulty: we generate synthetic training pairs by selecting segments in an image and copy-pasting them into another image. We then learn to predict the repeated region masks. We find that it is crucial to predict the correspondences as an auxiliary task and to use Poisson blending and style transfer on the training pairs to generalize on real data. We analyse results with two deep architectures relevant to our joint image analysis task: a transformer-based architecture and Sparse Nc-Net, a recent network designed to predict coarse correspondences using 4D convolutions. We show our approach provides clear improvements for artwork details retrieval on the Brueghel dataset and achieves competitive performance on two place recognition benchmarks, Tokyo247 and Pitts30K. We also demonstrate the potential of our approach for unsupervised image collection analysis by introducing a spectral graph clustering approach to object discovery and demonstrating it on the object discovery dataset of citerubinstein2013unsupervised and the Brueghel dataset. Our code and data are available at http://imagine.enpc.fr/~shenx/SegSwap/.