mlsanityData to 14 Oct 2026

Contrastive Learning with Stronger Augmentations

arXiv 2104.07713 ↗First posted 15 Apr 2021cs.CV cs.AI cs.LGcited by 100

Not on the map. Why?

Abstract (arXiv)

Representation learning has significantly been developed with the advance of contrastive learning methods. Most of those methods have benefited from various data augmentations that are carefully designated to maintain their identities so that the images transformed from the same instance can still be retrieved. However, those carefully designed transformations limited us to further explore the novel patterns exposed by other transformations. Meanwhile, as found in our experiments, the strong augmentations distorted the images' structures, resulting in difficult retrieval. Thus, we propose a general framework called Contrastive Learning with Stronger Augmentations~(CLSA) to complement current contrastive learning approaches. Here, the distribution divergence between the weakly and strongly augmented images over the representation bank is adopted to supervise the retrieval of strongly augmented queries from a pool of instances. Experiments on the ImageNet dataset and downstream datasets showed the information from the strongly augmented images can significantly boost the performance. For example, CLSA achieves top-1 accuracy of 76.2% on ImageNet with a standard ResNet-50 architecture with a single-layer classifier fine-tuned, which is almost the same level as 76.5% of supervised results. The code and pre-trained models are available in https://github.com/maple-research-lab/CLSA.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Builds on 49 of its 69 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

49 of 69 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
37 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  2. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  3. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  4. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  5. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  6. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  7. 2019Contrastive Multiview Codingcited by 1,301
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  9. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  10. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  11. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  12. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  13. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  14. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  15. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  16. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 657
  17. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  18. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  19. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 370
  20. 2016Adversarial Feature Learningcited by 350
  21. 2017Data Augmentation Generative Adversarial Networkscited by 278
  22. 2019Fast AutoAugmentcited by 243
  23. 2019Large Scale Adversarial Representation Learningcited by 239
  24. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  25. 2020Representation Learning via Invariant Causal Mechanismscited by 199
  26. 2019Population Based Augmentation: Efficient Learning of Augmentation Policy Schedulescited by 154
  27. 2019Classification Accuracy Score for Conditional Generative Modelscited by 136
  28. 2019Faster AutoAugment: Learning Augmentation Strategies using Backpropagationcited by 96
  29. 2020AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversariescited by 90
  30. 2019AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Datacited by 77
  31. 2020Learning Representations by Predicting Bags of Visual Wordscited by 73
  32. 2020OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learningcited by 65
  33. 2021Spatially Consistent Representation Learningcited by 64
  34. 2020Unsupervised Feature Learning by Cross-Level Instance-Group Discriminationcited by 51
  35. 2020CO2: Consistent Contrast for Unsupervised Visual Representation Learningcited by 38
  36. 2019Learning Generalized Transformation Equivariant Representations via Autoencoding Transformationscited by 22
  37. 2019EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformationscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 23; 2023: 27; 2024: 21; 2025: 13; 2026: 3.

100 papers.

Sort:

By area:

  1. 2021Parametric Contrastive Learningcited by 217
  2. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  3. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  4. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  5. 2021Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognitioncited by 90
  6. 2021ReSSL: Relational Self-Supervised Learning with Weak Augmentationcited by 84
  7. 2021Equivariant Contrastive Learningcited by 66
  8. 2022Twin Contrastive Learning for Online Clusteringcited by 60
  9. 2022Generalized Parametric Contrastive Learningcited by 45
  10. 2022HCSC: Hierarchical Contrastive Selective Codingcited by 43
  11. 2022On the Importance of Asymmetry for Siamese Representation Learningcited by 39
  12. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
38 more
  1. 2022Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentationscited by 31
  2. 2022Min-Max Similarity: A Contrastive Semi-Supervised Deep Learning Network for Surgical Tools Segmentationcited by 28
  3. 2023Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualizationcited by 25
  4. 2022Facial Video-based Remote Physiological Measurement via Self-supervised Learningcited by 23
  5. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  6. 2023Advancing Volumetric Medical Image Segmentation via Global-Local Masked Autoencodercited by 21
  7. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learningcited by 20
  8. 2023AutoProSAM: Automated Prompting SAM for 3D Multi-Organ Segmentationcited by 19
  9. 2024Self-supervised Learning for Electroencephalogram: A Systematic Surveycited by 19
  10. 2021Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learningcited by 18
  11. 2022A Coding Framework and Benchmark towards Low-Bitrate Video Understandingcited by 17
  12. 2022Strongly Augmented Contrastive Clusteringcited by 14
  13. 2023Unsupervised Representation Learning for Time Series: A Reviewcited by 14
  14. 2022Adaptive Soft Contrastive Learningcited by 13
  15. 2023Towards Enhancing Time Series Contrastive Learning: A Dynamic Bad Pair Mining Approachcited by 13
  16. 2022Analysis of Augmentations for Contrastive ECG Representation Learningcited by 12
  17. 2021Directional Self-supervised Learning for Heavy Image Augmentationscited by 11
  18. 2022CaCo: Both Positive and Negative Samples are Directly Learnable via Cooperative-adversarial Contrastive Learningcited by 9
  19. 2023Modulate Your Spectrum in Self-Supervised Learningcited by 9
  20. 2024Poly-View Contrastive Learningcited by 9
  21. 2022RenyiCL: Contrastive Representation Learning with Skew Renyi Divergencecited by 8
  22. 2022Semantics-Consistent Feature Search for Self-Supervised Visual Representation Learningcited by 8
  23. 2023MAST: Masked Augmentation Subspace Training for Generalizable Self-Supervised Priorscited by 8
  24. 2023Mini-Batch Optimization of Contrastive Losscited by 8
  25. 2024Domain-Specific Block Selection and Paired-View Pseudo-Labeling for Online Test-Time Adaptationcited by 8
  26. 2021A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learningcited by 7
  27. 2023No Free Lunch in Self Supervised Representation Learningcited by 7
  28. 2023DiffAug: Enhance Unsupervised Contrastive Learning with Domain-Knowledge-Free Diffusion-based Data Augmentationcited by 7
  29. 2024Exploring Correlations of Self-Supervised Tasks for Graphscited by 7
  30. 2022GEDI: GEnerative and DIscriminative Training for Self-Supervised Learningcited by 6
  31. 2023Contrastive Learning with Consistent Representationscited by 6
  32. 2023Relational Contrastive Learning for Scene Text Recognitioncited by 6
  33. 2021Group Contrastive Self-Supervised Learning on Graphscited by 5
  34. 2021Normality-Calibrated Autoencoder for Unsupervised Anomaly Detection on Data Contaminationcited by 5
  35. 2024SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learningcited by 5
  36. 2025Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clientscited by 5
  37. 2022Learning Behavior Representations Through Multi-Timescale Bootstrappingcited by 4
  38. 2022Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learningcited by 4

50 most cited of 100 All 100 (JSON)