# UniVIP: A Unified Framework for Self-Supervised Visual Pre-training

arXiv: https://arxiv.org/abs/2203.06965 · HTML: https://mlsanity.com/paper/2203.06965/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 51 references that are in the map.
- First posted on arXiv 14 Mar 2022.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [DetCo: Unsupervised Contrastive Learning for Object Detection](https://mlsanity.com/paper/2102.04803/) (2021)
- [Scaling and Benchmarking Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/1905.01235/) (2019)
- [OTA: Optimal Transport Assignment for Object Detection](https://mlsanity.com/paper/2103.14259/) (2021)
- [Unsupervised Learning of Dense Visual Representations](https://mlsanity.com/paper/2011.05499/) (2020)
- [Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases](https://mlsanity.com/paper/2007.13916/) (2020)
- [Self-Supervised Learning with Swin Transformers](https://mlsanity.com/paper/2105.04553/) (2021)
- [MST: Masked Self-Supervised Transformer for Visual Representation](https://mlsanity.com/paper/2106.05656/) (2021)

## Built on by (most cited first, 30 of 31)

- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2304.06461/) (2023)
- [Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks](https://mlsanity.com/paper/2209.13948/) (2022)
- [Self-supervised Facial Action Unit Detection with Region and Relation Learning](https://mlsanity.com/paper/2303.05708/) (2023)
- [Correlational Image Modeling for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2303.12670/) (2023)
- [Understanding Hyperbolic Metric Learning through Hard Negative Sampling](https://mlsanity.com/paper/2404.15523/) (2024)
- [A Shapelet-based Framework for Unsupervised Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2305.18888/) (2023)
- [Semantics-Consistent Feature Search for Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2212.06486/) (2022)
- [ProtoCLIP: Prototypical Contrastive Language Image Pretraining](https://mlsanity.com/paper/2206.10996/) (2022)
- [Transfering Low-Frequency Features for Domain Adaptation](https://mlsanity.com/paper/2208.14706/) (2022)
- [Semantic Contrastive Bootstrapping for Single-positive Multi-label Recognition](https://mlsanity.com/paper/2307.07680/) (2023)
- [Revisiting DETR Pre-training for Object Detection](https://mlsanity.com/paper/2308.01300/) (2023)
- [PAD: Self-Supervised Pre-Training with Patchwise-Scale Adapter for Infrared Images](https://mlsanity.com/paper/2312.08192/) (2023)
- [Composite Learning for Robust and Effective Dense Predictions](https://mlsanity.com/paper/2210.07239/) (2022)
- [LeOCLR: Leveraging Original Images for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2403.06813/) (2024)
- [Semantic Concentration for Self-Supervised Dense Representations Learning](https://mlsanity.com/paper/2509.09429/) (2025)
- [Efficient Masked Autoencoders with Self-Consistency](https://mlsanity.com/paper/2302.14431/) (2023)
- [Semantic Positive Pairs for Enhancing Visual Representation Learning of Instance Discrimination Methods](https://mlsanity.com/paper/2306.16122/) (2023)
- [Multi-Grained Contrast for Data-Efficient Unsupervised Representation Learning](https://mlsanity.com/paper/2407.02014/) (2024)
- [Ladder Siamese Network: a Method and Insights for Multi-level Self-Supervised Learning](https://mlsanity.com/paper/2211.13844/) (2022)
- [Multi-Level Contrastive Learning for Dense Prediction Task](https://mlsanity.com/paper/2304.02010/) (2023)
- [POA: Pre-training Once for Models of All Sizes](https://mlsanity.com/paper/2408.01031/) (2024)
- [FreConv: Frequency Branch-and-Integration Convolutional Networks](https://mlsanity.com/paper/2304.04540/) (2023)
- [Aligned Unsupervised Pretraining of Object Detectors with Self-training](https://mlsanity.com/paper/2307.15697/) (2023)
- [Beyond the Encoder: Joint Encoder-Decoder Contrastive Pre-Training Improves Dense Prediction](https://mlsanity.com/paper/2503.17526/) (2025)
- [SimMIL: A Universal Weakly Supervised Pre-Training Framework for Multi-Instance Learning in Whole Slide Pathology Images](https://mlsanity.com/paper/2505.06710/) (2025)
- [Multiple Object Stitching for Unsupervised Representation Learning](https://mlsanity.com/paper/2506.07364/) (2025)
- [UNIV: Unified Foundation Model for Infrared and Visible Modalities](https://mlsanity.com/paper/2509.15642/) (2025)
- [VINO: Video-driven Invariance for Non-contextual Objects via Structural Prior Guided De-contextualization](https://mlsanity.com/paper/2603.07222/) (2026)
- [Adapting Dense Vision-Language Relationships for Multi-label Classification with Partial Label](https://mlsanity.com/paper/2608.22313/) (2026)

## Abstract (arXiv)

Self-supervised learning (SSL) holds promise in leveraging large amounts of unlabeled data. However, the success of popular SSL methods has limited on single-centric-object images like those in ImageNet and ignores the correlation among the scene and instances, as well as the semantic difference of instances in the scene. To address the above problems, we propose a Unified Self-supervised Visual Pre-training (UniVIP), a novel self-supervised framework to learn versatile visual representations on either single-centric-object or non-iconic dataset. The framework takes into account the representation learning at three levels: 1) the similarity of scene-scene, 2) the correlation of scene-instance, 3) the discrimination of instance-instance. During the learning, we adopt the optimal transport algorithm to automatically measure the discrimination of instances. Massive experiments show that UniVIP pre-trained on non-iconic COCO achieves state-of-the-art transfer performance on a variety of downstream tasks, such as image classification, semi-supervised learning, object detection and segmentation. Furthermore, our method can also exploit single-centric-object dataset such as ImageNet and outperforms BYOL by 2.5% with the same pre-training epochs in linear probing, and surpass current self-supervised object detection methods on COCO dataset, demonstrating its universality and potential.