Data to 14 Oct 2026

Self-Supervised Learning of Pretext-Invariant Representations

arXiv 1912.01991 ↗First posted 4 Dec 2019cs.CV cs.LGcited by 808

Not on the map. Why?

Abstract (arXiv)

The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations for a large training set of images. Many pretext tasks lead to representations that are covariant with image transformations. We argue that, instead, semantic representations ought to be invariant under such transformations. Specifically, we develop Pretext-Invariant Representation Learning (PIRL, pronounced as "pearl") that learns invariant representations based on pretext tasks. We use PIRL with a commonly used pretext task that involves solving jigsaw puzzles. We find that PIRL substantially improves the semantic quality of the learned image representations. Our approach sets a new state-of-the-art in self-supervised learning from images on several popular benchmarks for self-supervised learning. Despite being unsupervised, PIRL outperforms supervised pre-training in learning image representations for object detection. Altogether, our results demonstrate the potential of self-supervised learning of image representations with good invariance properties.

Key facts. Cited by 808 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 251; 2022: 226; 2023: 162; 2024: 94; 2025: 53; 2026: 22.

808 papers.

Sort:

By area:

2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
2021Robust fine-tuning of zero-shot modelscited by 797
2021Model-Contrastive Federated Learningcited by 766
2022SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagerycited by 408
2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 326
2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313

38 more

2022Masked Siamese Networks for Label-Efficient Learningcited by 302
2021Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentationcited by 268
2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
2022Visual Prompting via Image Inpaintingcited by 250
2023A Cookbook of Self-Supervised Learningcited by 250
2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 215
2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
2021DetCo: Unsupervised Contrastive Learning for Object Detectioncited by 205
2021Self-supervised Pretraining of Visual Features in the Wildcited by 198
2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
2023Cut and Learn for Unsupervised Object Detection and Instance Segmentationcited by 187
2021Partial success in closing the gap between human and machine visioncited by 185
2021BEVT: BERT Pretraining of Video Transformerscited by 169
2021Benchmarking Representation Learning for Natural World Image Collectionscited by 168
2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 163
2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 156
2021VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examplescited by 154
2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
2021Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Datacited by 152
2022Benchmarking Self-Supervised Learning on Diverse Pathology Datasetscited by 150
2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
2021Decoupled Contrastive Learningcited by 143
2021Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizationcited by 141
2022Self-supervised Learning in Remote Sensing: A Reviewcited by 137
2021TCLR: Temporal Contrastive Learning for Video Representationcited by 136
2021SEED: Self-supervised Distillation For Visual Representationcited by 134
2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
2022What to Hide from Your Students: Attention-Guided Masked Image Modelingcited by 118
2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115

50 most cited of 808 All 808 (JSON)