mlsanityData to 7 Oct 2026

Informed Learning by Wide Neural Networks: Convergence, Generalization and Sampling Complexity

arXiv 2207.00751 ↗First posted 2 Jul 2022cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

By integrating domain knowledge with labeled samples, informed machine learning has been emerging to improve the learning performance for a wide range of applications. Nonetheless, rigorous understanding of the role of injected domain knowledge has been under-explored. In this paper, we consider an informed deep neural network (DNN) with over-parameterization and domain knowledge integrated into its training objective function, and study how and why domain knowledge benefits the performance. Concretely, we quantitatively demonstrate the two benefits of domain knowledge in informed learning - regularizing the label-based supervision and supplementing the labeled samples - and reveal the trade-off between label and knowledge imperfectness in the bound of the population risk. Based on the theoretical analysis, we propose a generalized informed training objective to better exploit the benefits of knowledge and balance the label and knowledge imperfectness, which is validated by the population risk bound. Our analysis on sampling complexity sheds lights on how to choose the hyper-parameters for informed learning, and further justifies the advantages of knowledge informed learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 61 of its 101 references that are in the map. First posted on arXiv 2 Jul 2022.

Builds on

61 of 101 references.

References · 61

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2020Knowledge Distillation: A Surveycited by 1,578
  4. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  5. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  7. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  8. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  9. 2019DeepXDE: A deep learning library for solving differential equationscited by 617
  10. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  11. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  12. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
38 more
  1. 2018Born Again Neural Networkscited by 539
  2. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  3. 2019On the Efficacy of Knowledge Distillationcited by 377
  4. 2017Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equationscited by 362
  5. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  6. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  7. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  8. 2017A Semantic Loss Function for Deep Learning with Symbolic Knowledgecited by 267
  9. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  10. 2019Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systemscited by 203
  11. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  12. 2021Towards Understanding Knowledge Distillationcited by 158
  13. 2020Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representationscited by 154
  14. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  15. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  16. 2017Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modelingcited by 130
  17. 2019A Primer on PAC-Bayesian Learningcited by 130
  18. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  19. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  20. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  21. 2017Learning to Optimize: Training Deep Neural Networks for Wireless Resource Managementcited by 96
  22. 2020Augmenting Physical Models with Deep Networks for Complex Dynamics Forecastingcited by 93
  23. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  24. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  25. 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
  26. 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
  27. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  28. 2016The More You Know: Using Knowledge Graphs for Image Classificationcited by 63
  29. 2017Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equationscited by 62
  30. 2017Hyperparameter Importance Across Datasetscited by 62
  31. 2019Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networkscited by 60
  32. 2018Towards Optimal Power Control via Ensembling Deep Neural Networkscited by 38
  33. 2019Wireless Networks Design in the Era of Deep Learning: Model-Based, AI-Based, or Both?cited by 38
  34. 2018One-Shot Unsupervised Cross Domain Translationcited by 35
  35. 2018Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networkscited by 35
  36. 2020Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teachercited by 33
  37. 2018Spatial Deep Learning for Wireless Schedulingcited by 32
  38. 2020On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type PDEscited by 31

50 most cited of 61 All 61 (JSON)

Built on by

Not cited yet.