# Self-supervised Pre-training with Hard Examples Improves Visual Representations

arXiv: https://arxiv.org/abs/2012.13493 · HTML: https://mlsanity.com/paper/2012.13493/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Dec 2020.

## Built on by (most cited first, 12 of 12)

- [On Feature Decorrelation in Self-Supervised Learning](https://mlsanity.com/paper/2105.00470/) (2021)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [Recent advances and clinical applications of deep learning in medical image analysis](https://mlsanity.com/paper/2105.13381/) (2021)
- [Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Training](https://mlsanity.com/paper/2305.13689/) (2023)
- [A Simple Data Mixing Prior for Improving Self-Supervised Learning](https://mlsanity.com/paper/2206.07692/) (2022)
- [Contrastive Attraction and Contrastive Repulsion for Representation Learning](https://mlsanity.com/paper/2105.03746/) (2021)
- [Improving Self-supervised Learning with Hardness-aware Dynamic Curriculum Learning: An Application to Digital Pathology](https://mlsanity.com/paper/2108.07183/) (2021)
- [Understanding Hyperbolic Metric Learning through Hard Negative Sampling](https://mlsanity.com/paper/2404.15523/) (2024)
- [Learning crop type mapping from regional label proportions in large-scale SAR and optical imagery](https://mlsanity.com/paper/2208.11607/) (2022)
- [Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data](https://mlsanity.com/paper/2104.02932/) (2021)
- [LeOCLR: Leveraging Original Images for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2403.06813/) (2024)
- [Contrastive pretraining for semantic segmentation is robust to noisy positive pairs](https://mlsanity.com/paper/2211.13756/) (2022)

## Abstract (arXiv)

Self-supervised pre-training (SSP) employs random image transformations to generate training data for visual representation learning. In this paper, we first present a modeling framework that unifies existing SSP methods as learning to predict pseudo-labels. Then, we propose new data augmentation methods of generating training examples whose pseudo-labels are harder to predict than those generated via random image transformations. Specifically, we use adversarial training and CutMix to create hard examples (HEXA) to be used as augmented views for MoCo-v2 and DeepCluster-v2, leading to two variants HEXA_{MoCo} and HEXA_{DCluster}, respectively. In our experiments, we pre-train models on ImageNet and evaluate them on multiple public benchmarks. Our evaluation shows that the two new algorithm variants outperform their original counterparts, and achieve new state-of-the-art on a wide range of tasks where limited task supervision is available for fine-tuning. These results verify that hard examples are instrumental in improving the generalization of the pre-trained models.