mlsanityData to 7 Oct 2026

Provable Benefit of Cutout and CutMix for Feature Learning

arXiv 2410.23672 ↗First posted 31 Oct 2024cs.LG cs.AI stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Patch-level data augmentation techniques such as Cutout and CutMix have demonstrated significant efficacy in enhancing the performance of vision tasks. However, a comprehensive theoretical understanding of these methods remains elusive. In this paper, we study two-layer neural networks trained using three distinct methods: vanilla training without augmentation, Cutout training, and CutMix training. Our analysis focuses on a feature-noise data model, which consists of several label-dependent features of varying rarity and label-independent noises of differing strengths. Our theorems demonstrate that Cutout training can learn low-frequency features that vanilla training cannot, while CutMix training can learn even rarer features that Cutout cannot capture. From this, we establish that CutMix yields the highest test accuracy among the three. Our novel analysis reveals that CutMix training makes the network learn all features and noise vectors "evenly" regardless of the rarity and strength, which provides an interesting insight into understanding patch-level augmentation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

31 of 42 references.

References · 31

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  4. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  5. 2020Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixupcited by 260
  6. 2020How Does Mixup Help With Robustness and Generalization?cited by 193
  7. 2021Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversitycited by 127
  8. 2020GridMask Data Augmentationcited by 123
  9. 2018A Kernel Theory of Modern Data Augmentationcited by 106
  10. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  11. 2020On Mixup Regularizationcited by 82
  12. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
19 more
  1. 2021When and How Mixup Improves Calibrationcited by 59
  2. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48
  3. 2020On the Generalization Effects of Linear Transformations in Data Augmentationcited by 47
  4. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  5. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  6. 2022A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspectivecited by 27
  7. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  8. 2021Towards Understanding the Data Dependency of Mixup-style Trainingcited by 19
  9. 2023The Benefits of Mixup for Feature Learningcited by 19
  10. 2019Does Data Augmentation Lead to Positive Margin?cited by 18
  11. 2022Data Augmentation as Feature Manipulationcited by 18
  12. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18
  13. 2020How Data Augmentation affects Optimization for Linear Regressioncited by 15
  14. 2019A Group-Theoretic Framework for Data Augmentationcited by 14
  15. 2022Provably Learning Diverse Features in Multi-View Data with Midpoint Mixupcited by 14
  16. 2023Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptronscited by 12
  17. 2023Provable Benefit of Mixup for Finding Optimal Decision Boundariescited by 6
  18. 2023On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamicscited by 5
  19. 2024For Better or For Worse? Learning Minimum Variance Features With Label Augmentationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025Reperio-rPPG: Relational Temporal Graph Neural Networks for Periodicity Learning in Remote Physiological Measurementcited by 3
  2. 2025Enabling Validation for Robust Few-Shot Recognitioncited by 2
  3. 2025NSegment : Label-specific Deformations for Remote Sensing Image Segmentationcited by 0
  4. 2026The Impact of CutMix on Reliability and Robustness in Semantic Segmentationcited by 0