# On the Properties of Kullback-Leibler Divergence Between Multivariate Gaussian Distributions

arXiv: https://arxiv.org/abs/2102.05485 · HTML: https://mlsanity.com/paper/2102.05485/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 82 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 20 of 20)

- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization](https://mlsanity.com/paper/1606.00709/) (2016)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [Why Normalizing Flows Fail to Detect Out-of-Distribution Data](https://mlsanity.com/paper/2006.08545/) (2020)
- [An elementary introduction to information geometry](https://mlsanity.com/paper/1808.08271/) (2018)
- [Optimal Bounds between f-Divergences and Integral Probability Metrics](https://mlsanity.com/paper/2006.05973/) (2020)
- [Practical and Consistent Estimation of f-Divergences](https://mlsanity.com/paper/1905.11112/) (2019)
- [A Less Biased Evaluation of Out-of-distribution Sample Detectors](https://mlsanity.com/paper/1809.04729/) (2018)
- [Minimax Optimal Estimation of KL Divergence for Continuous Distributions](https://mlsanity.com/paper/2002.11599/) (2020)
- [(f,Gamma)-Divergences: Interpolating between f-Divergences and Integral Probability Metrics](https://mlsanity.com/paper/2011.05953/) (2020)
- [Online Markov decision processes with Kullback-Leibler control cost](https://mlsanity.com/paper/1401.3198/) (2014)
- [Multivariate f-Divergence Estimation With Confidence](https://mlsanity.com/paper/1411.2045/) (2014)
- [Are generative deep models for novelty detection truly better?](https://mlsanity.com/paper/1807.05027/) (2018)
- [A New Family of Divergences Originating from Model Adequacy Tests and Application to Robust Statistical Inference](https://mlsanity.com/paper/1611.05224/) (2016)
- [Unifying computational entropies via Kullback-Leibler divergence](https://mlsanity.com/paper/1902.11202/) (2019)
- [Toward a generic representation of random variables for machine learning](https://mlsanity.com/paper/1506.00976/) (2015)

## Built on by (most cited first, 23 of 23)

- [Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding](https://mlsanity.com/paper/2503.09143/) (2025)
- [Embedding Safety into RL: A New Take on Trust Region Methods](https://mlsanity.com/paper/2411.02957/) (2024)
- [PPGFlowECG: Latent Rectified Flow with Cross-Modal Encoding for PPG-Guided ECG Generation and Cardiovascular Disease Detection](https://mlsanity.com/paper/2509.19774/) (2025)
- [Hallucination Detection: A Probabilistic Framework Using Embeddings Distance Analysis](https://mlsanity.com/paper/2502.08663/) (2025)
- [Ensemble Kalman filter in latent space using a variational autoencoder pair](https://mlsanity.com/paper/2502.12987/) (2025)
- [Temporal Source Recovery for Time-Series Source-Free Unsupervised Domain Adaptation](https://mlsanity.com/paper/2409.19635/) (2024)
- [The Statistical Fairness-Accuracy Frontier](https://mlsanity.com/paper/2508.17622/) (2025)
- [Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression](https://mlsanity.com/paper/2502.10587/) (2025)
- [AdaptFuse: Training-Free Sequential Preference Inference via Externalized Bayesian Computation](https://mlsanity.com/paper/2604.03925/) (2026)
- [Video-Robin: Autoregressive Diffusion Planning for Intent-Grounded Video-to-Music Generation](https://mlsanity.com/paper/2604.17656/) (2026)
- [Bit Efficient Toeplitz Covariance Estimation](https://mlsanity.com/paper/2412.12678/) (2024)
- [Nip Rumors in the Bud: Retrieval-Guided Topic-Level Adaptation for Test-Time Fake News Video Detection](https://mlsanity.com/paper/2601.11981/) (2026)
- [Relaxed Triangle Inequality for Kullback-Leibler Divergence Between Multivariate Gaussian Distributions](https://mlsanity.com/paper/2602.02577/) (2026)
- [Converging Dimensions: Information Extraction and Summarization through Multisource, Multimodal, and Multilingual Fusion](https://mlsanity.com/paper/2406.13715/) (2024)
- [RUMI: Rummaging Using Mutual Information](https://mlsanity.com/paper/2408.10450/) (2024)
- [Efficient Sample-optimal Learning of Gaussian Tree Models via Sample-optimal Testing of Gaussian Mutual Information](https://mlsanity.com/paper/2411.11516/) (2024)
- [Self-supervised denoising of visual field data improves detection of glaucoma progression](https://mlsanity.com/paper/2411.12146/) (2024)
- [Multi-Source Unsupervised Domain Adaptation with Prototype Aggregation](https://mlsanity.com/paper/2412.16255/) (2024)
- [Monte Carlo Functional Regularisation for Continual Learning](https://mlsanity.com/paper/2508.13006/) (2025)
- [Optimal Stability of KL Divergence under Gaussian Perturbations](https://mlsanity.com/paper/2604.11026/) (2026)
- [DCP-Prune: Ultra-Low Token Pruning with Distribution Consistency Preservation](https://mlsanity.com/paper/2606.16633/) (2026)
- [VIBE: Video Instruction-aligned Background music gEneration](https://mlsanity.com/paper/2608.30125/) (2026)
- [FERPO: Forward Entropy-Regularized Policy Optimization](https://mlsanity.com/paper/2610.02198/) (2026)

## Abstract (arXiv)

Kullback-Leibler (KL) divergence is one of the most important divergence measures between probability distributions. In this paper, we prove several properties of KL divergence between multivariate Gaussian distributions. First, for any two n-dimensional Gaussian distributions N1 and N2, we give the supremum of KL(N1||N2) when KL(N2||N1)≤ varepsilon (varepsilon>0). For small varepsilon, we show that the supremum is varepsilon + 2varepsilon1.5 + O(varepsilon2). This quantifies the approximate symmetry of small KL divergence between Gaussians. We also find the infimum of KL(N1||N2) when KL(N2||N1)≥ M (M>0). We give the conditions when the supremum and infimum can be attained. Second, for any three n-dimensional Gaussians N1, N2, and N3, we find an upper bound of KL(N1||N3) if KL(N1||N2)≤ varepsilon1 and KL(N2||N3)≤ varepsilon2 for varepsilon1,varepsilon2≥ 0. For small varepsilon1 and varepsilon2, we show the upper bound is 3varepsilon1+3varepsilon2+2√varepsilon1varepsilon2+o(varepsilon1)+o(varepsilon2). This reveals that KL divergence between Gaussians follows a relaxed triangle inequality. Importantly, all the bounds in the theorems presented in this paper are independent of the dimension n. Finally, We discuss the applications of our theorems in explaining counterintuitive phenomenon of flow-based model, deriving deep anomaly detection algorithm, and extending one-step robustness guarantee to multiple steps in safe reinforcement learning.