mlsanityData to 7 Oct 2026

Generalization bounds via distillation

arXiv 2104.05641 ↗First posted 12 Apr 2021cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

This paper theoretically investigates the following empirical phenomenon: given a high-complexity network with poor generalization bounds, one can distill it into a network with nearly identical predictions but low complexity and vastly smaller generalization bounds. The main contribution is an analysis showing that the original network inherits this good generalization bound from its distillation, assuming the use of well-behaved data augmentation. This bound is presented both in an abstract and in a concrete form, the latter complemented by a reduction technique to handle modern computation graphs featuring convolutional layers, fully-connected layers, and skip connections, to name a few. To round out the story, a (looser) classical uniform convergence analysis of compression is also presented, as well as a variety of experiments on cifar and mnist demonstrating similar generalization performance between the original network and its distillation.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

17 of 31 references.

References · 17

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  5. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  6. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  7. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  8. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  9. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  10. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  11. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  12. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 111
5 more
  1. 2020In Search of Robust Measures of Generalizationcited by 79
  2. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  3. 2020Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpotcited by 66
  4. 2019Generalization bounds for deep convolutional neural networkscited by 58
  5. 2019Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural networkcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 5; 2023: 8; 2024: 5; 2025: 5; 2026: 3.

33 papers.

Sort:

  1. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  2. 2021Heavy Tails in SGD and Compressibility of Overparametrized Neural Networkscited by 44
  3. 2023Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learningcited by 32
  4. 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
  5. 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
  6. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillationcited by 23
  7. 2022Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithmscited by 22
  8. 2021Self-training Converts Weak Learners to Strong Learners in Mixture Modelscited by 19
  9. 2023PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guaranteescited by 15
  10. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  11. 2024Minimum Description Length and Generalization Guarantees for Representation Learningcited by 11
  12. 2022The Sample Complexity of One-Hidden-Layer Neural Networkscited by 10
21 more
  1. 2021MixACM: Mixup-Based Robustness Transfer via Distillation of Activated Channel Mapscited by 9
  2. 2023Data-dependent Generalization Bounds via Variable-Size Compressibilitycited by 9
  3. 2023Learning From Biased Soft Labelscited by 8
  4. 2023Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clusteringcited by 7
  5. 2021On Predicting Generalization using GANscited by 6
  6. 2024How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learningcited by 6
  7. 2023Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGDcited by 5
  8. 2025Deep Learning as a Convex Paradigm of Computation: Minimizing Circuit Size with ResNetscited by 5
  9. 2023Towards the Fundamental Limits of Knowledge Transfer over Finite Domainscited by 4
  10. 2024Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatchcited by 4
  11. 2024Slicing Mutual Information Generalization Bounds for Neural Networkscited by 4
  12. 2024FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillationcited by 3
  13. 2025Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priorscited by 2
  14. 2026Bound to Disagree: Generalization Bounds via Certifiable Surrogatescited by 2
  15. 2025Multiview Representation Learning via Distributed Joint Latent Space Structuringcited by 1
  16. 2025Geometric Analysis of Neural Regression Collapse via Intrinsic Dimensioncited by 1
  17. 2021Compression Implies Generalizationcited by 0
  18. 2022Effectiveness of Function Matching in Driving Scene Recognitioncited by 0
  19. 2025Enriching Knowledge Distillation with Intra-Class Contrastive Learningcited by 0
  20. 2026Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networkscited by 0
  21. 2026On the Generalization of Knowledge Distillation: An Information-Theoretic Viewcited by 0