mlsanityData to 14 Oct 2026

Distributional Generalization: A New Kind of Generalization

arXiv 2009.08092 ↗First posted 17 Sep 2020cs.LG cs.NE math.STcited by 42

Not on the map. Why?

Abstract (arXiv)

We introduce a new notion of generalization -- Distributional Generalization -- which roughly states that outputs of a classifier at train and test time are close *as distributions*, as opposed to close in just their average error. For example, if we mislabel 30% of dogs as cats in the train set of CIFAR-10, then a ResNet trained to interpolation will in fact mislabel roughly 30% of dogs as cats on the *test set* as well, while leaving other classes unaffected. This behavior is not captured by classical generalization, which would only consider the average error and not the distribution of errors over the input domain. Our formal conjectures, which are much more general than this example, characterize the form of distributional generalization that can be expected in terms of problem parameters: model architecture, training procedure, number of samples, and data distribution. We give empirical evidence for these conjectures across a variety of domains in machine learning, including neural networks, kernel machines, and decision trees. Our results thus advance our empirical understanding of interpolating classifiers.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 13; 2023: 13; 2024: 5; 2025: 4; 2026: 2.

42 papers.

Sort:

By area:

  1. 2021Explaining Neural Scaling Lawscited by 363
  2. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  3. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  4. 2021Assessing Generalization of SGD via Disagreementcited by 117
  5. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  6. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 89
  7. 2022Towards Adversarial Evaluations for Inexact Machine Unlearningcited by 84
  8. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 60
  9. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  10. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  11. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  12. 2021Early-stopped neural networks are consistentcited by 34
30 more
  1. 2023On the Variance of Neural Network Training with respect to Test Sets and Distributionscited by 22
  2. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 21
  3. 2022Deconstructing Distributions: A Pointwise Framework of Learningcited by 19
  4. 2023More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatorycited by 18
  5. 2022A Note on "Assessing Generalization of SGD via Disagreement"cited by 16
  6. 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
  7. 2023(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancycited by 16
  8. 2022Knowledge Distillation: Bad Models Can Be Good Role Modelscited by 12
  9. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  10. 2022What You See is What You Get: Principled Deep Learning via Distributional Generalizationcited by 11
  11. 2023Conditioning Predictive Models: Risks and Strategiescited by 11
  12. 2025Monitoring Risks in Test-Time Adaptationcited by 11
  13. 2024Advancing Deep Active Learning & Data Subset Selection: Unifying Principles with Information-Theory Intuitionscited by 7
  14. 2024Predicting the Performance of Foundation Models via Agreement-on-the-Linecited by 7
  15. 2023Inconsistency, Instability, and Generalization Gap of Deep Neural Network Trainingcited by 6
  16. 2023Interpolation Learning With Minimum Description Lengthcited by 5
  17. 2023On the Joint Interaction of Models, Data, and Featurescited by 5
  18. 2023Unsupervised Accuracy Estimation of Deep Visual Models using Domain-Adaptive Adversarial Perturbation without Source Samplescited by 3
  19. 2024Machine unlearning through fine-grained model parameters perturbationcited by 2
  20. 2023Beyond Top-Class Agreement: Using Divergences to Forecast Performance under Distribution Shiftcited by 1
  21. 2024AdaRank: Disagreement Based Module Rank Prediction for Low-rank Adaptationcited by 1
  22. 2025Towards Better Generalization via Distributional Input Projection Networkcited by 1
  23. 2026Model Agreement via Anchoringcited by 1
  24. 2021Explaining generalization in deep learning: progress and fundamental limitscited by 0
  25. 2022Variation-based Cause Effect Identificationcited by 0
  26. 2023On the Efficacy of Generalization Error Prediction Scoring Functionscited by 0
  27. 2023Resource-constrained knowledge diffusion processes inspired by human peer learningcited by 0
  28. 2025ALSA: Anchors in Logit Space for Out-of-Distribution Accuracy Estimationcited by 0
  29. 2025Feature Dynamics as Implicit Data Augmentation: A Depth-Decomposed View on Deep Neural Network Generalizationcited by 0
  30. 2026Memorization in image classification: test set behavior and scaling lawscited by 0