Relaxed Triangle Inequality for Kullback-Leibler Divergence Between Multivariate Gaussian Distributions

The Kullback-Leibler (KL) divergence is not a proper distance metric and does not satisfy the triangle inequality, posing theoretical challenges in certain practical applications. Existing work has demonstrated that KL divergence between multivariate Gaussian distributions follows a relaxed triangle inequality. Given any three multivariate Gaussian distributions N1, N2, and N3, if KL(N1, N2)≤ ε1 and KL(N2, N3)≤ ε2, then KL(N1, N3)< 3ε1+3ε2+2√ε1ε2+o(ε1)+o(ε2). However, the supremum of KL(N1, N3) is still unknown. In this paper, we investigate the relaxed triangle inequality for the KL divergence between multivariate Gaussian distributions and give the supremum of KL(N1, N3) as well as the conditions when the supremum can be attained. When ε1 and ε2 are small, the supremum is ε1+ε2+2√ε1ε2+o(ε1)+o(ε2). Finally, we demonstrate several applications of our results in out-of-distribution detection with flow-based generative models and safe reinforcement learning.