# An Analysis of Posterior Collapse, Parameterization and Initialization in Variational Deep Gaussian Processes

arXiv: https://arxiv.org/abs/2606.25882 · HTML: https://mlsanity.com/paper/2606.25882/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 75 references that are in the map.
- First posted on arXiv 24 Jun 2026.

## Builds on (most cited first, 10 of 10)

- [A Framework for Interdomain and Multioutput Gaussian Processes](https://mlsanity.com/paper/2003.01115/) (2020)
- [A Tutorial on Sparse Gaussian Processes and Variational Inference](https://mlsanity.com/paper/2012.13962/) (2020)
- [GPflux: A Library for Deep Gaussian Processes](https://mlsanity.com/paper/2104.05674/) (2021)
- [Avoiding mode collapse in diffusion models fine-tuned with reinforcement learning](https://mlsanity.com/paper/2410.08315/) (2024)
- [Hierarchical Gaussian Processes with Wasserstein-2 Kernels](https://mlsanity.com/paper/2010.14877/) (2020)
- [Towards Flexibility and Interpretability of Gaussian Process State-Space Model](https://mlsanity.com/paper/2301.08843/) (2023)
- [Neural Likelihoods for Multi-Output Gaussian Processes](https://mlsanity.com/paper/1905.13697/) (2019)
- [Deep Gaussian Processes with Decoupled Inducing Inputs](https://mlsanity.com/paper/1801.02939/) (2018)
- [Deep Transformed Gaussian Processes](https://mlsanity.com/paper/2310.18230/) (2023)
- [The Jacobian and Hessian of the Kullback-Leibler Divergence between Multivariate Gaussian Distributions (Technical Report)](https://mlsanity.com/paper/2506.23996/) (2025)

## Abstract (arXiv)

DGPs are probabilistic models with remarkable prediction performance that concatenate GPs across several layers. Exact inference in DGPs is intractable, and variational inference is often used to approximate the posterior with a parametric distribution tuned by minimizing the Kullback-Leibler divergence. Moreover, finding a good VI approximation is challenging. In particular, a problem of VI is posterior collapse, where VI converges to a variational posterior that matches the prior. In variational DGPs, this implies explaining the data as noise. This work studies posterior collapse in DGPs and identifies its connection to the DSVI algorithm and the widely used linear prior mean function employed in all but the last layer. We show that the benefit of the linear prior mean does not arise from avoiding the non-injective pathology in very deep DGPs, as previously believed, but from improving the conditioning of the optimization problem at initialization. Thus, we propose an alternative initialization of a zero prior mean DGP that mimics a DGP with a linear prior mean at initialization. This enables successful training of DGPs without imposing optimization-driven constraints on the prior, allowing to choose the prior based on modeling assumptions rather than optimization convenience. Our analysis considers three common parameterizations of DGPs and shows that not all of them benefit from a linear prior mean. We also explain why a whitened parameterization of the DGP provides more stable convergence, something often assumed from experience, but lacking a rigorous analysis. Furthermore, we show that this stability is also beneficial to avoid the posterior collapse problem. Extensive experiments validate our findings: the proposed initialization prevents posterior collapse, improves stability, and achieves performance comparable to (and sometimes better than) DGPs with a linear prior mean.