# Fine-Grained Self-Supervised Learning with Jigsaw Puzzles for Medical Image Classification

arXiv: https://arxiv.org/abs/2308.05770 · HTML: https://mlsanity.com/paper/2308.05770/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 45 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches](https://mlsanity.com/paper/2003.03836/) (2020)
- [Self-supervised Feature Learning for 3D Medical Images by Playing a Rubik's Cube](https://mlsanity.com/paper/1910.02241/) (2019)
- [Learning Image Representations by Completing Damaged Jigsaw Puzzles](https://mlsanity.com/paper/1802.01880/) (2018)
- [Channel Interaction Networks for Fine-Grained Image Categorization](https://mlsanity.com/paper/2003.05235/) (2020)
- [Self-Loop Uncertainty: A Novel Pseudo-Label for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2007.09854/) (2020)
- [ProCo: Prototype-aware Contrastive Learning for Long-tailed Medical Image Classification](https://mlsanity.com/paper/2209.00183/) (2022)
- [ACT: Semi-supervised Domain-adaptive Medical Image Segmentation with Asymmetric Co-training](https://mlsanity.com/paper/2206.02288/) (2022)
- [A Novel Multi-task Deep Learning Model for Skin Lesion Segmentation and Classification](https://mlsanity.com/paper/1703.01025/) (2017)
- [Data-Driven Color Augmentation Techniques for Deep Skin Image Analysis](https://mlsanity.com/paper/1703.03702/) (2017)
- [SimTriplet: Simple Triplet Representation Learning with a Single GPU](https://mlsanity.com/paper/2103.05585/) (2021)
- [Fine-grained Retrieval Prompt Tuning](https://mlsanity.com/paper/2207.14465/) (2022)
- [Incorporating the Knowledge of Dermatologists to Convolutional Neural Networks for the Diagnosis of Skin Lesions](https://mlsanity.com/paper/1703.01976/) (2017)
- [Distractor-Aware Neuron Intrinsic Learning for Generic 2D Medical Image Classifications](https://mlsanity.com/paper/2007.09979/) (2020)
- [Evaluating histopathology transfer learning with ChampKit](https://mlsanity.com/paper/2206.06862/) (2022)

## Built on by (most cited first, 7 of 7)

- [CXR-LT 2024: A MICCAI challenge on long-tailed, multi-label, and zero-shot disease classification from chest X-ray](https://mlsanity.com/paper/2506.07984/) (2025)
- [Self-Supervised Learning for Image Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2505.13584/) (2025)
- [Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient Ultra-Fine-Grained Image Recognition](https://mlsanity.com/paper/2409.11051/) (2024)
- [Efficient Deep Learning Approaches for Processing Ultra-Widefield Retinal Imaging](https://mlsanity.com/paper/2503.18151/) (2025)
- [Attention-Based Chaotic Self-Supervision for Medical Image Classification](https://mlsanity.com/paper/2605.04985/) (2026)
- [Knowing When Not to Predict: Self Supervised Learning and Abstention for Safer DR Screening](https://mlsanity.com/paper/2605.19133/) (2026)
- [Chaos-SSL: An Attention-Based Self-Supervised Learning Framework with Chaotic Transformation for Medical Image Classification](https://mlsanity.com/paper/2605.27146/) (2026)

## Abstract (arXiv)

Classifying fine-grained lesions is challenging due to minor and subtle differences in medical images. This is because learning features of fine-grained lesions with highly minor differences is very difficult in training deep neural networks. Therefore, in this paper, we introduce Fine-Grained Self-Supervised Learning(FG-SSL) method for classifying subtle lesions in medical images. The proposed method progressively learns the model through hierarchical block such that the cross-correlation between the fine-grained Jigsaw puzzle and regularized original images is close to the identity matrix. We also apply hierarchical block for progressive fine-grained learning, which extracts different information in each step, to supervised learning for discovering subtle differences. Our method does not require an asymmetric model, nor does a negative sampling strategy, and is not sensitive to batch size. We evaluate the proposed fine-grained self-supervised learning method on comprehensive experiments using various medical image recognition datasets. In our experiments, the proposed method performs favorably compared to existing state-of-the-art approaches on the widely-used ISIC2018, APTOS2019, and ISIC2017 datasets.