mlsanityData to 14 Oct 2026

Gaussian Process Surrogate Models for Neural Networks

arXiv 2208.06028 ↗First posted 11 Aug 2022cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Not being able to understand and predict the behavior of deep learning systems makes it hard to decide what architecture and algorithm to use for a given problem. In science and engineering, modeling is a methodology used to understand complex systems whose internal processes are opaque. Modeling replaces a complex system with a simpler, more interpretable surrogate. Drawing inspiration from this, we construct a class of surrogate models for neural networks using Gaussian processes. Rather than deriving kernels for infinite neural networks, we learn kernels empirically from the naturalistic behavior of finite neural networks. We demonstrate our approach captures existing phenomena related to the spectral bias of neural networks, and then show that our surrogate models can be used to solve practical problems such as identifying which points most influence the behavior of specific neural networks and predicting which architectures and algorithms will generalize well for specific datasets.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 51 of its 82 references that are in the map. First posted on arXiv 11 Aug 2022.

Builds on

51 of 82 references.

References · 51

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2013Intriguing properties of neural networkscited by 5,155
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  9. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  11. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  12. 2018On the Spectral Bias of Neural Networkscited by 1,341
38 more
  1. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  2. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  3. 2020Meta-Learning in Neural Networks: A Surveycited by 950
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  5. 2016The Mythos of Model Interpretabilitycited by 721
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  7. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  8. 2017Deep Neural Networks as Gaussian Processescited by 622
  9. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  10. 2017Exploring Generalization in Deep Learningcited by 600
  11. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  12. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  13. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  14. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 222
  15. 2017Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Modelscited by 206
  16. 2016Deep Information Propagationcited by 195
  17. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  18. 2021The Principles of Deep Learning Theorycited by 173
  19. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  20. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  21. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 149
  22. 2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
  23. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
  24. 2021The Benchmark Lotterycited by 136
  25. 2013Gaussian Process Kernels for Pattern Discovery and Extrapolationcited by 132
  26. 2019Exact Gaussian Processes on a Million Data Pointscited by 119
  27. 2016Why does deep and cheap learning work so well?cited by 100
  28. 2013Structure Discovery in Nonparametric Regression through Compositional Kernel Searchcited by 88
  29. 2020The Unreasonable Effectiveness of Deep Learning in Artificial Intelligencecited by 57
  30. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 57
  31. 2017Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Studycited by 46
  32. 2020NeurIPS 2020 Competition: Predicting Generalization in Deep Learningcited by 41
  33. 2014Avoiding pathologies in very deep networkscited by 33
  34. 2017Theory of Deep Learning III: explaining the non-overfitting puzzlecited by 32
  35. 2019Approximate Inference for Fully Bayesian Gaussian Process Regressioncited by 31
  36. 2020Investigating representations of verb bias in neural language modelscited by 22
  37. 2022ExSum: From Local Explanations to Model Understandingcited by 15
  38. 2021Kernel Identification Through Transformerscited by 13

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 3.

5 papers.

  1. 2023Bayes in the age of intelligent machinescited by 22
  2. 2024Feature Fusion for Improved Classification: Combining Dempster-Shafer Theory and Multiple CNN Architecturescited by 0
  3. 2025BoxRL-NNV: Boxed Refinement of Latin Hypercube Samples for Neural Network Verificationcited by 0
  4. 2025A Surrogate Model for the Forward Design of Multi-layered Metasurface-based Radar Absorbing Structurescited by 0
  5. 2025A Modality-Aware Cooperative Co-Evolutionary Framework for Multimodal Graph Neural Architecture Searchcited by 0