mlsanityData to 7 Oct 2026

Avoiding pathologies in very deep networks

arXiv 1402.5836 ↗First posted 24 Feb 2014stat.ML cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Choosing appropriate architectures and regularization strategies for deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous problem of constructing useful priors on compositions of functions. Specifically, we study the deep Gaussian process, a type of infinitely-wide, deep neural network. We show that in standard architectures, the representational capacity of the network tends to capture fewer degrees of freedom as the number of layers increases, retaining only a single degree of freedom in the limit. We propose an alternate network architecture which does not suffer from this pathology. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Lastly, we characterize the class of models obtained by performing dropout on Gaussian processes.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Feb 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 7; 2023: 5; 2024: 6; 2025: 3; 2026: 2.

33 papers.

Sort:

  1. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  2. 2022The Unreasonable Effectiveness of Deep Evidential Regressioncited by 40
  3. 2021Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equationscited by 38
  4. 2022Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifierscited by 28
  5. 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
  6. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 26
  7. 2021Multivariate Deep Evidential Regressioncited by 23
  8. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  9. 2022Statistical Deep Learning for Spatial and Spatio-Temporal Datacited by 15
  10. 2021Deep Gaussian Process Emulation using Stochastic Imputationcited by 11
  11. 2021Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Featurescited by 8
  12. 2022Deep Variational Implicit Processescited by 8
21 more
  1. 2021A variational approximate posterior for the deep Wishart processcited by 6
  2. 2022Gaussian Process Surrogate Models for Neural Networkscited by 5
  3. 2024Fully Bayesian Differential Gaussian Processes through Stochastic Differential Equationscited by 4
  4. 2023Thin and Deep Gaussian Processescited by 3
  5. 2024Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selectioncited by 3
  6. 2022Deep Layer-wise Networks Have Closed-Form Weightscited by 2
  7. 2024Functional Stochastic Gradient MCMC for Bayesian Neural Networkscited by 2
  8. 2021Conditional Deep Gaussian Processes: empirical Bayes hyperdata learningcited by 1
  9. 2022On Connecting Deep Trigonometric Networks with Deep Gaussian Processes: Covariance, Expressivity, and Neural Tangent Kernelcited by 1
  10. 2023Bayesian inference with finitely wide neural networkscited by 1
  11. 2023Spatial Bayesian Neural Networkscited by 1
  12. 2023Deep Latent Force Models: ODE-based Process Convolutions for Bayesian Deep Learningcited by 1
  13. 2024Adaptive RKHS Fourier Features for Compositional Gaussian Process Modelscited by 1
  14. 2024Amortized Variational Inference for Deep Gaussian Processescited by 1
  15. 2021Deep limits and cut-off phenomena for neural networkscited by 0
  16. 2024Deep Manifold Part 1: Anatomy of Neural Network Manifoldcited by 0
  17. 2025Support Collapse of Deep Gaussian Processes with Polynomial Kernels for a Wide Regime of Hyperparameterscited by 0
  18. 2025Sparse Techniques for Regression in Deep Gaussian Processescited by 0
  19. 2025Simulation Priors for Data-Efficient Deep Learningcited by 0
  20. 2026How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPscited by 0
  21. 2026Recursive Gaussian Processes and the Bayesian Braincited by 0