# Region Similarity Representation Learning

arXiv: https://arxiv.org/abs/2103.12902 · HTML: https://mlsanity.com/paper/2103.12902/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 58 references that are in the map.
- First posted on arXiv 24 Mar 2021.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Learning Representations by Maximizing Mutual Information Across Views](https://mlsanity.com/paper/1906.00910/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)

## Built on by (most cited first, 30 of 96)

- [Benchmarking Self-Supervised Learning on Diverse Pathology Datasets](https://mlsanity.com/paper/2212.04690/) (2022)
- [VICRegL: Self-Supervised Learning of Local Visual Features](https://mlsanity.com/paper/2210.01571/) (2022)
- [Aligning Pretraining for Detection via Object-Level Contrastive Learning](https://mlsanity.com/paper/2106.02637/) (2021)
- [FreeSOLO: Learning to Segment Objects without Annotations](https://mlsanity.com/paper/2202.12181/) (2022)
- [BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations](https://mlsanity.com/paper/2201.04684/) (2022)
- [Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2305.07011/) (2023)
- [Class-agnostic Object Detection with Multi-modal Transformer](https://mlsanity.com/paper/2111.11430/) (2021)
- [DETReg: Unsupervised Pretraining with Region Priors for Object Detection](https://mlsanity.com/paper/2106.04550/) (2021)
- [Unsupervised Object-Level Representation Learning from Scene Images](https://mlsanity.com/paper/2106.11952/) (2021)
- [Patch-level Representation Learning for Self-supervised Vision Transformers](https://mlsanity.com/paper/2206.07990/) (2022)
- [LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching](https://mlsanity.com/paper/2306.11925/) (2023)
- [MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features](https://mlsanity.com/paper/2307.12698/) (2023)
- [Green Hierarchical Vision Transformer for Masked Image Modeling](https://mlsanity.com/paper/2205.13515/) (2022)
- [Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations](https://mlsanity.com/paper/2106.05967/) (2021)
- [CMID: A Unified Self-Supervised Learning Framework for Remote Sensing Image Understanding](https://mlsanity.com/paper/2304.09670/) (2023)
- [Masked Image Modeling with Denoising Contrast](https://mlsanity.com/paper/2205.09616/) (2022)
- [SimPer: Simple Self-Supervised Learning of Periodic Targets](https://mlsanity.com/paper/2210.03115/) (2022)
- [Point-Level Region Contrast for Object Detection Pre-Training](https://mlsanity.com/paper/2202.04639/) (2022)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [A Survey of Self-Supervised and Few-Shot Object Detection](https://mlsanity.com/paper/2110.14711/) (2021)
- [UniVIP: A Unified Framework for Self-Supervised Visual Pre-training](https://mlsanity.com/paper/2203.06965/) (2022)
- [Learning Where to Learn in Cross-View Self-Supervised Learning](https://mlsanity.com/paper/2203.14898/) (2022)
- [Architecture-Agnostic Masked Image Modeling -- From ViT back to CNN](https://mlsanity.com/paper/2205.13943/) (2022)
- [Contrastive Feature Masking Open-Vocabulary Vision Transformer](https://mlsanity.com/paper/2309.00775/) (2023)
- [Exploring Set Similarity for Dense Self-supervised Representation Learning](https://mlsanity.com/paper/2107.08712/) (2021)
- [CrOC: Cross-View Online Clustering for Dense Visual Representation Learning](https://mlsanity.com/paper/2303.13245/) (2023)
- [Location-Aware Self-Supervised Transformers for Semantic Segmentation](https://mlsanity.com/paper/2212.02400/) (2022)
- [Top-Down Visual Attention from Analysis by Synthesis](https://mlsanity.com/paper/2303.13043/) (2023)
- [Multi-Label Self-Supervised Learning with Scene Images](https://mlsanity.com/paper/2308.03286/) (2023)
- [Panoramic Panoptic Segmentation: Insights Into Surrounding Parsing for Mobile Agents via Unsupervised Contrastive Learning](https://mlsanity.com/paper/2206.10711/) (2022)

## Abstract (arXiv)

We present Region Similarity Representation Learning (ReSim), a new approach to self-supervised representation learning for localization-based tasks such as object detection and segmentation. While existing work has largely focused on solely learning global representations for an entire image, ReSim learns both regional representations for localization as well as semantic image-level representations. ReSim operates by sliding a fixed-sized window across the overlapping area between two views (e.g., image crops), aligning these areas with their corresponding convolutional feature map regions, and then maximizing the feature similarity across views. As a result, ReSim learns spatially and semantically consistent feature representation throughout the convolutional feature maps of a neural network. A shift or scale of an image region, e.g., a shift or scale of an object, has a corresponding change in the feature maps; this allows downstream tasks to leverage these representations for localization. Through object detection, instance segmentation, and dense pose estimation experiments, we illustrate how ReSim learns representations which significantly improve the localization and classification performance compared to a competitive MoCo-v2 baseline: +2.7 APbb75 VOC, +1.1 APbb75 COCO, and +1.9 APmk Cityscapes. Code and pre-trained models are released at: urlhttps://github.com/Tete-Xiao/ReSim