mlsanityData to 14 Oct 2026

Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning

arXiv 2410.11206 ↗First posted 15 Oct 2024cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Semi-supervised learning (SSL), exemplified by FixMatch (Sohn et al., 2020), has shown significant generalization advantages over supervised learning (SL), particularly in the context of deep neural networks (DNNs). However, it is still unclear, from a theoretical standpoint, why FixMatch-like SSL algorithms generalize better than SL on DNNs. In this work, we present the first theoretical justification for the enhanced test accuracy observed in FixMatch-like SSL applied to DNNs by taking convolutional neural networks (CNNs) on classification tasks as an example. Our theoretical analysis reveals that the semantic feature learning processes in FixMatch and SL are rather different. In particular, FixMatch learns all the discriminative features of each semantic class, while SL only randomly captures a subset of features due to the well-known lottery ticket hypothesis. Furthermore, we show that our analysis framework can be applied to other FixMatch-like SSL methods, e.g., FlexMatch, FreeMatch, Dash, and SoftMatch. Inspired by our theoretical analysis, we develop an improved variant of FixMatch, termed Semantic-Aware FixMatch (SA-FixMatch). Experimental results corroborate our theoretical findings and the enhanced generalization capability of SA-FixMatch.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

37 of 55 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2016Wide Residual Networkscited by 3,070
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  11. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
25 more
  1. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  2. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  3. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
  4. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  5. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  6. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  7. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  8. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  9. 2014Learning with Pseudo-Ensemblescited by 241
  10. 2022FreeMatch: Self-adaptive Thresholding for Semi-supervised Learningcited by 232
  11. 2020fastai: A Layered API for Deep Learningcited by 183
  12. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 180
  13. 2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 148
  14. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  15. 2023SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learningcited by 133
  16. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  17. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  18. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  19. 2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
  20. 2017Adversarial Dropout for Supervised and Semi-supervised Learningcited by 39
  21. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  22. 2022Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscited by 18
  23. 2021Information-Theoretic Characterization of the Generalization Error for Iterative Semi-Supervised Learningcited by 11
  24. 2022Don't fear the unlabelled: safe semi-supervised learning via simple debiasingcited by 10
  25. 2023Can semi-supervised learning use all the data effectively? A lower bound perspectivecited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025SpikeMatch: Semi-Supervised Learning with Temporal Dynamics of Spiking Neural Networkscited by 0
  2. 2026Learning from Label Proportions with Dual-proportion Constraintscited by 0
  3. 2026Semi-Supervised Noise Adaptation: Transferring Knowledge from Noise Domaincited by 0
  4. 2026Learning Dynamics of Logits Debiasing for Long-Tailed Semi-Supervised Learningcited by 0
  5. 2026A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilizationcited by 0