On Connecting Deep Trigonometric Networks with Deep Gaussian Processes: Covariance, Expressivity, and Neural Tangent Kernel
Not on the map. Why?
Abstract (arXiv)
Deep Gaussian Process (DGP) as a model prior in Bayesian learning intuitively exploits the expressive power in function composition. DGPs also offer diverse modeling capabilities, but inference is challenging because marginalization in latent function space is not tractable. With Bochner's theorem, DGP with squared exponential kernel can be viewed as a deep trigonometric network consisting of the random feature layers, sine and cosine activation units, and random weight layers. In the wide limit with a bottleneck, we show that the weight space view yields the same effective covariance functions which were obtained previously in function space. Also, varying the prior distributions over network parameters is equivalent to employing different kernels. As such, DGPs can be translated into the deep bottlenecked trig networks, with which the exact maximum a posteriori estimation can be obtained. Interestingly, the network representation enables the study of DGP's neural tangent kernel, which may also reveal the mean of the intractable predictive distribution. Statistically, unlike the shallow networks, deep networks of finite width have covariance deviating from the limiting kernel, and the inner and outer widths may play different roles in feature learning. Numerical simulations are present to support our findings.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 50 of its 72 references that are in the map. First posted on arXiv 14 Mar 2022.
Builds on
50 of 72 references.
References · 50
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
- 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
- 2016Understanding deep learning requires rethinking generalizationcited by 1,657
- 2017DOTA: A Large-scale Dataset for Object Detection in Aerial Imagescited by 622
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
- 2017CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learningcited by 459
- 2015Deep Kernel Learningcited by 440
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 314
- 2016Exponential expressivity in deep neural networks through transient chaoscited by 260
38 more
- 2016Deep Information Propagationcited by 194
- 2021The Principles of Deep Learning Theorycited by 169
- 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
- 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
- 2017Doubly Stochastic Variational Inference for Deep Gaussian Processescited by 143
- 2016Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivitycited by 138
- 2013Gaussian Process Kernels for Pattern Discovery and Extrapolationcited by 131
- 2019Finite Depth and Width Corrections to the Neural Tangent Kernelcited by 114
- 2016Orthogonal Random Featurescited by 89
- 2019Approximate Inference Turns Deep Networks into Gaussian Processescited by 89
- 2013Structure Discovery in Nonparametric Regression through Compositional Kernel Searchcited by 86
- 2021The Promises and Pitfalls of Deep Kernel Learningcited by 81
- 2019Non-Gaussian processes and neural networks at finite widthscited by 78
- 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
- 2019Deep Gaussian Processes for Multi-fidelity Modelingcited by 48
- 2020Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processescited by 43
- 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
- 2016Deep Gaussian Processes for Regression using Approximate Expectation Propagationcited by 39
- 2014Avoiding pathologies in very deep networkscited by 33
- 2020Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?cited by 33
- 2016Random Feature Expansions for Deep Gaussian Processescited by 32
- 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
- 2018Differentiable Compositional Kernel Learning for Gaussian Processescited by 23
- 2019Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functionscited by 22
- 2019Finite size corrections for neural network Gaussian processescited by 22
- 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
- 2019Deep Gaussian Processes with Importance-Weighted Variational Inferencecited by 18
- 2020Wide Neural Networks with Bottlenecks are Deep Gaussian Processescited by 18
- 2019Implicit Posterior Variational Inference for Deep Gaussian Processescited by 17
- 2019Function-Space Distributions over Kernelscited by 17
- 2020Physics Informed Deep Kernel Learningcited by 12
- 2015Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputscited by 11
- 2019Interpretable deep Gaussian processes with momentscited by 8
- 2019Compositional uncertainty in deep Gaussian processescited by 8
- 2021Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Featurescited by 8
- 2020Conditional Deep Gaussian Processes: multi-fidelity kernel learningcited by 3
- 2020Characterizing Deep Gaussian Processes via Nonlinear Recurrence Systemscited by 2
- 2021Conditional Deep Gaussian Processes: empirical Bayes hyperdata learningcited by 1
Built on by
Citing papers per year: 2024: 1.
1 paper.