mlsanityData to 7 Oct 2026

Measuring Generalization with Optimal Transport

arXiv 2106.03314 ↗First posted 7 Jun 2021cs.LG stat.MLcited by 24

Not on the map. Why?

Abstract (arXiv)

Understanding the generalization of deep neural networks is one of the most important tasks in deep learning. Although much progress has been made, theoretical error bounds still often behave disparately from empirical observations. In this work, we develop margin-based generalization bounds, where the margins are normalized with optimal transport costs between independent random subsets sampled from the training distribution. In particular, the optimal transport cost can be interpreted as a generalization of variance which captures the structural properties of the learned feature space. Our bounds robustly predict the generalization error, given training data and network parameters, on large scale datasets. Theoretically, we demonstrate that the concentration and separation of features play crucial roles in generalization, supporting empirical results in the literature. The code is available at urlhttps://github.com/chingyaoc/kV-Margin.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

41 of 62 references.

References · 41

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2016Wide Residual Networkscited by 3,057
  6. 2020Supervised Contrastive Learningcited by 2,803
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  11. 2020Contrastive Learning with Hard Negative Samplescited by 619
  12. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
29 more
  1. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  2. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  3. 2020Debiased Contrastive Learningcited by 354
  4. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 309
  5. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  6. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  7. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  8. 2017Size-Independent Sample Complexity of Neural Networkscited by 289
  9. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  10. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  11. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 182
  12. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  13. 2018Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networkscited by 140
  14. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 111
  15. 2018Large Margin Deep Networks for Classificationcited by 110
  16. 2020Exactly Computing the Local Lipschitz Constant of ReLU Networkscited by 102
  17. 2019ClusterFit: Improving Generalization of Visual Representationscited by 82
  18. 2020In Search of Robust Measures of Generalizationcited by 79
  19. 2019Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margincited by 56
  20. 2020Estimating Generalization under Distribution Shifts via Domain-Invariant Representationscited by 56
  21. 2018Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent Constraintscited by 53
  22. 2019Generalization in Deep Networks: The Role of Distance from Initializationcited by 49
  23. 2020NeurIPS 2020 Competition: Predicting Generalization in Deep Learningcited by 41
  24. 2020Representation Based Complexity Measures for Predicting Generalization in Deep Learningcited by 29
  25. 2020Separation and Concentration in Deep Networkscited by 28
  26. 2021A Dimension-free Computational Upper-bound for Smooth Optimal Transport Estimationcited by 17
  27. 2020k-Variance: A Clustered Notion of Variancecited by 11
  28. 2020Ranking Deep Learning Generalization using Label Variation in Latent Geometry Graphscited by 6
  29. 2021Gi and Pal Scores: Deep Neural Network Generalization Statisticscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 6; 2024: 6; 2025: 5; 2026: 1.

24 papers.

Sort:

By area:

  1. 2022Robust Contrastive Learning against Noisy Viewscited by 45
  2. 2023Scalable Optimal Transport Methods in Machine Learning: A Contemporary Surveycited by 27
  3. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  4. 2023Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimationcited by 22
  5. 2025X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usabilitycited by 20
  6. 2022Integral Probability Metrics PAC-Bayes Boundscited by 19
  7. 2024Towards Bridging Generalization and Expressivity of Graph Neural Networkscited by 12
  8. 2023Sample Complexity Bounds for Estimating Probability Divergences under Invariancescited by 11
  9. 2022On Measuring Excess Capacity in Neural Networkscited by 9
  10. 2022InfoOT: Information Maximizing Optimal Transportcited by 9
  11. 2024What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?cited by 6
  12. 2023Input margins can predict generalization toocited by 4
12 more
  1. 2024Slicing Mutual Information Generalization Bounds for Neural Networkscited by 4
  2. 2021In Search of Probeable Generalization Measurescited by 2
  3. 2024Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancingcited by 2
  4. 2025Beyond External Monitors: Enhancing Transparency of Large Language Models for Easier Monitoringcited by 2
  5. 2025Data-Agnostic Augmentations for Unknown Variations: Out-of-Distribution Generalisation in MRI Segmentationcited by 2
  6. 2025Understanding Generalization in Node and Link Predictioncited by 2
  7. 2022Meet You Halfway: Explaining Deep Learning Mysteriescited by 0
  8. 2023Generalization within in silico screeningcited by 0
  9. 2024Twin: Tuning Learning Rate and Weight Decay of Deep Homogeneous Classifiers without Validationcited by 0
  10. 2024Generalization of Graph Neural Networks through the Lens of Homomorphismcited by 0
  11. 2025Bispectral OT: Dataset Comparison using Symmetry-Aware Optimal Transportcited by 0
  12. 2026Towards Initialization-dependent and Non-vacuous Generalization Bounds for Overparameterized Shallow Neural Networkscited by 0