# Jigsaw Clustering for Unsupervised Visual Representation Learning

arXiv: https://arxiv.org/abs/2104.00323 · HTML: https://mlsanity.com/paper/2104.00323/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 1 Apr 2021.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [S4L: Self-Supervised Semi-Supervised Learning](https://mlsanity.com/paper/1905.03670/) (2019)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Large Scale Adversarial Representation Learning](https://mlsanity.com/paper/1907.02544/) (2019)
- [Local Aggregation for Unsupervised Learning of Visual Embeddings](https://mlsanity.com/paper/1903.12355/) (2019)
- [Learning Features by Watching Objects Move](https://mlsanity.com/paper/1612.06370/) (2016)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)

## Built on by (most cited first, 30 of 30)

- [Self-supervised Learning in Remote Sensing: A Review](https://mlsanity.com/paper/2206.13188/) (2022)
- [Crafting Better Contrastive Views for Siamese Representation Learning](https://mlsanity.com/paper/2202.03278/) (2022)
- [Large-Scale 3D Medical Image Pre-training with Geometric Context Priors](https://mlsanity.com/paper/2410.09890/) (2024)
- [Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding](https://mlsanity.com/paper/2202.13093/) (2022)
- [Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works](https://mlsanity.com/paper/2211.08129/) (2022)
- [You Only Cut Once: Boosting Data Augmentation with a Single Cut](https://mlsanity.com/paper/2201.12078/) (2022)
- [Advancing Volumetric Medical Image Segmentation via Global-Local Masked Autoencoder](https://mlsanity.com/paper/2306.08913/) (2023)
- [MiM: Mask in Mask Self-Supervised Pre-Training for 3D Medical Image Analysis](https://mlsanity.com/paper/2404.15580/) (2024)
- [Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches](https://mlsanity.com/paper/2207.08220/) (2022)
- [Hallucination Improves the Performance of Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2307.12168/) (2023)
- [CropMix: Sampling a Rich Input Distribution via Multi-Scale Cropping](https://mlsanity.com/paper/2205.15955/) (2022)
- [Out-of-Distribution Detection via Deep Multi-Comprehension Ensemble](https://mlsanity.com/paper/2403.16260/) (2024)
- [Tailoring Self-Supervision for Supervised Learning](https://mlsanity.com/paper/2207.10023/) (2022)
- [Adaptive Data Augmentation for Contrastive Learning](https://mlsanity.com/paper/2304.02451/) (2023)
- [Clustering for Protein Representation Learning](https://mlsanity.com/paper/2404.00254/) (2024)
- [Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition](https://mlsanity.com/paper/2503.18746/) (2025)
- [Structure-aware Semantic Discrepancy and Consistency for 3D Medical Image Self-supervised Learning](https://mlsanity.com/paper/2507.02581/) (2025)
- [Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learning](https://mlsanity.com/paper/2211.01796/) (2022)
- [A Study of Forward-Forward Algorithm for Self-Supervised Learning](https://mlsanity.com/paper/2309.11955/) (2023)
- [Progressive Proxy Anchor Propagation for Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2407.12463/) (2024)
- [Unsupervised Local Discrimination for Medical Images](https://mlsanity.com/paper/2108.09440/) (2021)
- [A Feature Consistency Driven Attention Erasing Network for Fine-Grained Image Retrieval](https://mlsanity.com/paper/2110.04479/) (2021)
- [HOME: High-Order Mixed-Moment-based Embedding for Representation Learning](https://mlsanity.com/paper/2207.07743/) (2022)
- [On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning](https://mlsanity.com/paper/2404.19289/) (2024)
- [MedCL: Learning Consistent Anatomy Distribution for Scribble-supervised Medical Image Segmentation](https://mlsanity.com/paper/2503.22890/) (2025)
- [VisTopics: A Visual Semantic Unsupervised Approach to Topic Modeling of Video and Image Data](https://mlsanity.com/paper/2505.14868/) (2025)
- [Self-Supervised Modality-Aware Multiple Granularity Pre-Training for RGB-Infrared Person Re-Identification](https://mlsanity.com/paper/2112.06147/) (2021)
- [Multi-network Contrastive Learning Based on Global and Local Representations](https://mlsanity.com/paper/2306.15930/) (2023)
- [Grid Jigsaw Representation with CLIP: A New Perspective on Image Clustering](https://mlsanity.com/paper/2310.17869/) (2023)
- [Curriculum Multi-Task Self-Supervision Improves Lightweight Architectures for Onboard Satellite Hyperspectral Image Segmentation](https://mlsanity.com/paper/2509.13229/) (2025)

## Abstract (arXiv)

Unsupervised representation learning with contrastive learning achieved great success. This line of methods duplicate each training batch to construct contrastive pairs, making each training batch and its augmented version forwarded simultaneously and leading to additional computation. We propose a new jigsaw clustering pretext task in this paper, which only needs to forward each training batch itself, and reduces the training cost. Our method makes use of information from both intra- and inter-images, and outperforms previous single-batch based ones by a large margin. It is even comparable to the contrastive learning methods when only half of training batches are used. Our method indicates that multiple batches during training are not necessary, and opens the door for future research of single-batch unsupervised methods. Our models trained on ImageNet datasets achieve state-of-the-art results with linear classification, outperforming previous single-batch methods by 2.6%. Models transferred to COCO datasets outperform MoCo v2 by 0.4% with only half of the training batches. Our pretrained models outperform supervised ImageNet pretrained models on CIFAR-10 and CIFAR-100 datasets by 0.9% and 4.1% respectively. Code is available at https://github.com/Jia-Research-Lab/JigsawClustering