# A Simple Data Mixing Prior for Improving Self-Supervised Learning

arXiv: https://arxiv.org/abs/2206.07692 · HTML: https://mlsanity.com/paper/2206.07692/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 48 references that are in the map.
- First posted on arXiv 15 Jun 2022.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [iBOT: Image BERT Pre-Training with Online Tokenizer](https://mlsanity.com/paper/2111.07832/) (2021)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Hard Negative Mixing for Contrastive Learning](https://mlsanity.com/paper/2010.01028/) (2020)
- [Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup](https://mlsanity.com/paper/2009.06962/) (2020)

## Built on by (most cited first, 27 of 27)

- [Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognition](https://mlsanity.com/paper/2303.10904/) (2023)
- [Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentation](https://mlsanity.com/paper/2210.17400/) (2022)
- [SG-Former: Self-guided Transformer with Evolving Token Reallocation](https://mlsanity.com/paper/2308.12216/) (2023)
- [A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainability](https://mlsanity.com/paper/2212.10888/) (2022)
- [A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective](https://mlsanity.com/paper/2208.09913/) (2022)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learning](https://mlsanity.com/paper/2308.03975/) (2023)
- [RankMixup: Ranking-Based Mixup Training for Network Calibration](https://mlsanity.com/paper/2308.11990/) (2023)
- [How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?](https://mlsanity.com/paper/2501.11253/) (2025)
- [Understanding Hyperbolic Metric Learning through Hard Negative Sampling](https://mlsanity.com/paper/2404.15523/) (2024)
- [Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?](https://mlsanity.com/paper/2406.10743/) (2024)
- [Inter-Instance Similarity Modeling for Contrastive Learning](https://mlsanity.com/paper/2306.12243/) (2023)
- [Gen-SIS: Generative Self-augmentation Improves Self-supervised Learning](https://mlsanity.com/paper/2412.01672/) (2024)
- [Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Head](https://mlsanity.com/paper/2302.10260/) (2023)
- [Infinite Class Mixup](https://mlsanity.com/paper/2305.10293/) (2023)
- [Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation](https://mlsanity.com/paper/2308.04725/) (2023)
- [Unleash Model Potential: Bootstrapped Meta Self-supervised Learning](https://mlsanity.com/paper/2308.14267/) (2023)
- [Supervised Contrastive Learning on Blended Images for Long-tailed Recognition](https://mlsanity.com/paper/2211.11938/) (2022)
- [POA: Pre-training Once for Models of All Sizes](https://mlsanity.com/paper/2408.01031/) (2024)
- [Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Data](https://mlsanity.com/paper/2509.08469/) (2025)
- [SAMix: Calibrated and Accurate Continual Learning via Sphere-Adaptive Mixup and Neural Collapse](https://mlsanity.com/paper/2510.15751/) (2025)
- [A Self supervised learning framework for imbalanced medical imaging datasets](https://mlsanity.com/paper/2604.01947/) (2026)
- [Contrastive pretraining for semantic segmentation is robust to noisy positive pairs](https://mlsanity.com/paper/2211.13756/) (2022)
- [Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2306.01929/) (2023)
- [Representation Synthesis by Probabilistic Many-Valued Logic Operation in Self-Supervised Learning](https://mlsanity.com/paper/2309.04148/) (2023)
- [Adversarial Semi-Supervised Domain Adaptation for Semantic Segmentation: A New Role for Labeled Target Samples](https://mlsanity.com/paper/2312.07370/) (2023)
- [PreMix: Label-Efficient Multiple Instance Learning via Non-Contrastive Pre-training and Feature Mixing](https://mlsanity.com/paper/2408.01162/) (2024)

## Abstract (arXiv)

Data mixing (e.g., Mixup, Cutmix, ResizeMix) is an essential component for advancing recognition models. In this paper, we focus on studying its effectiveness in the self-supervised setting. By noticing the mixed images that share the same source images are intrinsically related to each other, we hereby propose SDMP, short for Simple Data Mixing Prior, to capture this straightforward yet essential prior, and position such mixed images as additional positive pairs to facilitate self-supervised representation learning. Our experiments verify that the proposed SDMP enables data mixing to help a set of self-supervised learning frameworks (e.g., MoCo) achieve better accuracy and out-of-distribution robustness. More notably, our SDMP is the first method that successfully leverages data mixing to improve (rather than hurt) the performance of Vision Transformers in the self-supervised setting. Code is publicly available at https://github.com/OliverRensu/SDMP