mlsanityData to 7 Oct 2026

Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles

arXiv 1603.09246 ↗First posted 30 Mar 2016cs.CVcited by 1,379

Not on the map. Why?

Abstract (arXiv)

In this paper we study the problem of image representation learning without human annotation. By following the principles of self-supervision, we build a convolutional neural network (CNN) that can be trained to solve Jigsaw puzzles as a pretext task, which requires no manual labeling, and then later repurposed to solve object classification and detection. To maintain the compatibility across tasks we introduce the context-free network (CFN), a siamese-ennead CNN. The CFN takes image tiles as input and explicitly limits the receptive field (or context) of its early processing units to one tile at a time. We show that the CFN includes fewer parameters than AlexNet while preserving the same semantic learning capabilities. By training the CFN to solve Jigsaw puzzles, we learn both a feature mapping of object parts as well as their correct spatial arrangement. Our experimental evaluations show that the learned features capture semantically relevant content. Our proposed method for learning visual representations outperforms state of the art methods in several transfer learning benchmarks.

Key facts. Cited by 1,379 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 412; 2022: 394; 2023: 253; 2024: 165; 2025: 97; 2026: 58.

1,379 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2022Visual Prompt Tuningcited by 1,674
  5. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  6. 2021Transformers in Vision: A Surveycited by 1,009
  7. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2021Robust fine-tuning of zero-shot modelscited by 785
  10. 2021Domain Generalization: A Surveycited by 760
  11. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  12. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
38 more
  1. 2021Extract Free Dense Labels from CLIPcited by 580
  2. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  3. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  4. 2021CutPaste: Self-Supervised Learning for Anomaly Detection and Localizationcited by 529
  5. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  6. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  7. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  8. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  9. 2021Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysiscited by 418
  10. 2022SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentationcited by 397
  11. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  12. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  13. 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 342
  14. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  15. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  16. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  17. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  18. 2021Graph Self-Supervised Learning: A Surveycited by 289
  19. 2022Contrastive Test-Time Adaptationcited by 258
  20. 2023A Cookbook of Self-Supervised Learningcited by 248
  21. 2022Masked Visual Pre-training for Motor Controlcited by 246
  22. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  23. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  24. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  25. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  26. 2021One Million Scenes for Autonomous Driving: ONCE Datasetcited by 218
  27. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  28. 2021Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Datacited by 214
  29. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  30. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  31. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  32. 2021DetCo: Unsupervised Contrastive Learning for Object Detectioncited by 205
  33. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  34. 2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
  35. 2021Unsupervised Semantic Segmentation by Contrasting Object Mask Proposalscited by 190
  36. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  37. 2021Reinforcement Learning with Prototypical Representationscited by 179
  38. 2021BEVT: BERT Pretraining of Video Transformerscited by 169

50 most cited of 1,379 All 1,379 (JSON)