mlsanityData to 14 Oct 2026

A Simple Data Mixing Prior for Improving Self-Supervised Learning

arXiv 2206.07692 ↗First posted 15 Jun 2022cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

Data mixing (e.g., Mixup, Cutmix, ResizeMix) is an essential component for advancing recognition models. In this paper, we focus on studying its effectiveness in the self-supervised setting. By noticing the mixed images that share the same source images are intrinsically related to each other, we hereby propose SDMP, short for Simple Data Mixing Prior, to capture this straightforward yet essential prior, and position such mixed images as additional positive pairs to facilitate self-supervised representation learning. Our experiments verify that the proposed SDMP enables data mixing to help a set of self-supervised learning frameworks (e.g., MoCo) achieve better accuracy and out-of-distribution robustness. More notably, our SDMP is the first method that successfully leverages data mixing to improve (rather than hurt) the performance of Vision Transformers in the self-supervised setting. Code is publicly available at https://github.com/OliverRensu/SDMP

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

40 of 48 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
28 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  2. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  4. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  7. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  9. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  11. 2019Natural Adversarial Examplescited by 1,408
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  13. 2016Colorful Image Colorizationcited by 1,114
  14. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  15. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  16. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  17. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  18. 2020Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixupcited by 261
  19. 2017Multi-task Self-Supervised Visual Learningcited by 200
  20. 2020SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularizationcited by 149
  21. 2020Online Deep Clustering for Unsupervised Representation Learningcited by 100
  22. 2018Boosting Self-Supervised Learning via Knowledge Transfercited by 92
  23. 2020Towards Domain-Agnostic Contrastive Learningcited by 75
  24. 2020ResizeMix: Mixing Data with Preserved Object Information and True Labelscited by 69
  25. 2020Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learningcited by 62
  26. 2020MixCo: Mix-up Contrastive Learning for Visual Representationcited by 62
  27. 2020Self-supervised Pre-training with Hard Examples Improves Visual Representationscited by 12
  28. 2021Cut-Thumbnail: A Novel Data Augmentation for Convolutional Neural Networkcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 12; 2024: 6; 2025: 3; 2026: 1.

27 papers.

Sort:

By area:

  1. 2023Actionlet-Dependent Contrastive Learning for Unsupervised Skeleton-Based Action Recognitioncited by 42
  2. 2022Max Pooling with Vision Transformers reconciles class and shape in weakly supervised semantic segmentationcited by 41
  3. 2023SG-Former: Self-guided Transformer with Evolving Token Reallocationcited by 32
  4. 2022A Survey of Mix-based Data Augmentation: Taxonomy, Methods, Applications, and Explainabilitycited by 31
  5. 2022A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspectivecited by 27
  6. 2024A Survey on Mixup Augmentations and Beyondcited by 20
  7. 2023Prompted Contrast with Masked Motion Modeling: Towards Versatile 3D Action Representation Learningcited by 19
  8. 2023RankMixup: Ranking-Based Mixup Training for Network Calibrationcited by 19
  9. 2025How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?cited by 19
  10. 2024Understanding Hyperbolic Metric Learning through Hard Negative Samplingcited by 10
  11. 2024Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?cited by 8
  12. 2023Inter-Instance Similarity Modeling for Contrastive Learningcited by 6
15 more
  1. 2024Gen-SIS: Generative Self-augmentation Improves Self-supervised Learningcited by 5
  2. 2023Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self-Supervision, Reconstruction, Projector Headcited by 4
  3. 2023Infinite Class Mixupcited by 2
  4. 2023Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillationcited by 2
  5. 2023Unleash Model Potential: Bootstrapped Meta Self-supervised Learningcited by 2
  6. 2022Supervised Contrastive Learning on Blended Images for Long-tailed Recognitioncited by 1
  7. 2024POA: Pre-training Once for Models of All Sizescited by 1
  8. 2025Maximally Useful and Minimally Redundant: The Key to Self Supervised Learning for Imbalanced Datacited by 1
  9. 2025SAMix: Calibrated and Accurate Continual Learning via Sphere-Adaptive Mixup and Neural Collapsecited by 1
  10. 2026A Self supervised learning framework for imbalanced medical imaging datasetscited by 1
  11. 2022Contrastive pretraining for semantic segmentation is robust to noisy positive pairscited by 0
  12. 2023Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Workcited by 0
  13. 2023Representation Synthesis by Probabilistic Many-Valued Logic Operation in Self-Supervised Learningcited by 0
  14. 2023Adversarial Semi-Supervised Domain Adaptation for Semantic Segmentation: A New Role for Labeled Target Samplescited by 0
  15. 2024PreMix: Label-Efficient Multiple Instance Learning via Non-Contrastive Pre-training and Feature Mixingcited by 0