# Information Theory with Kernel Methods

arXiv: https://arxiv.org/abs/2202.08545 · HTML: https://mlsanity.com/paper/2202.08545/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 72 references that are in the map.
- First posted on arXiv 17 Feb 2022.

## Builds on (most cited first, 12 of 12)

- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [An Introduction to Matrix Concentration Inequalities](https://mlsanity.com/paper/1501.01571/) (2015)
- [Kernel Mean Embedding of Distributions: A Review and Beyond](https://mlsanity.com/paper/1605.09522/) (2016)
- [Breaking the Curse of Dimensionality with Convex Neural Networks](https://mlsanity.com/paper/1412.8690/) (2014)
- [Generalization Properties of Learning with Random Features](https://mlsanity.com/paper/1602.04474/) (2016)
- [On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions](https://mlsanity.com/paper/1502.06800/) (2015)
- [Less is More: Nystr"om Computational Regularization](https://mlsanity.com/paper/1507.04717/) (2015)
- [Finding Global Minima via Kernel Approximations](https://mlsanity.com/paper/2012.11978/) (2020)
- [A Dimension-free Computational Upper-bound for Smooth Optimal Transport Estimation](https://mlsanity.com/paper/2101.05380/) (2021)
- [Optimal rates of entropy estimation over Lipschitz balls](https://mlsanity.com/paper/1711.02141/) (2017)
- [Relating Leverage Scores and Density using Regularized Christoffel Functions](https://mlsanity.com/paper/1805.07943/) (2018)
- [Infinite-Dimensional Sums-of-Squares for Optimal Control](https://mlsanity.com/paper/2110.07396/) (2021)

## Built on by (most cited first, 30 of 37)

- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models](https://mlsanity.com/paper/2310.00902/) (2023)
- [Does Representation Matter? Exploring Intermediate Layers in Large Language Models](https://mlsanity.com/paper/2412.09563/) (2024)
- [Matrix Information Theory for Self-Supervised Learning](https://mlsanity.com/paper/2305.17326/) (2023)
- [Information Flow in Self-Supervised Learning](https://mlsanity.com/paper/2309.17281/) (2023)
- [Understanding Grokking Through A Robustness Viewpoint](https://mlsanity.com/paper/2311.06597/) (2023)
- [Convergence Rates for Non-Log-Concave Sampling and Log-Partition Estimation](https://mlsanity.com/paper/2303.03237/) (2023)
- [The Representation Jensen-Shannon Divergence](https://mlsanity.com/paper/2305.16446/) (2023)
- [Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning](https://mlsanity.com/paper/2411.02344/) (2024)
- [Sum-of-Squares Relaxations for Information Theory and Variational Inference](https://mlsanity.com/paper/2206.13285/) (2022)
- [The Information of Large Language Model Geometry](https://mlsanity.com/paper/2402.03471/) (2024)
- [Information Shapes Koopman Representation](https://mlsanity.com/paper/2510.13025/) (2025)
- [Auditing Differential Privacy in High Dimensions with the Kernel Quantum R'enyi Divergence](https://mlsanity.com/paper/2205.13941/) (2022)
- [A kernel-based analysis of Laplacian Eigenmaps](https://mlsanity.com/paper/2402.16481/) (2024)
- [Unveiling the Dynamics of Information Interplay in Supervised Learning](https://mlsanity.com/paper/2406.03999/) (2024)
- [Learning to Embed Distributions via Maximum Kernel Entropy](https://mlsanity.com/paper/2408.00549/) (2024)
- [Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergence](https://mlsanity.com/paper/2408.16543/) (2024)
- [The Maximum von Neumann Entropy Principle: Theory and Applications in Machine Learning](https://mlsanity.com/paper/2602.02117/) (2026)
- [Distributed Gradient Descent for Functional Learning](https://mlsanity.com/paper/2305.07408/) (2023)
- [Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learning](https://mlsanity.com/paper/2406.01435/) (2024)
- [Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matrices](https://mlsanity.com/paper/2410.17998/) (2024)
- [Kernel Embeddings and the Separation of Measure Phenomenon](https://mlsanity.com/paper/2505.04613/) (2025)
- [Exposing Diversity Bias in Deep Generative Models: Statistical Origins and Correction of Diversity Error](https://mlsanity.com/paper/2602.14682/) (2026)
- [Enhancing Kernel Flexibility via Learning Asymmetric Locally-Adaptive Kernels](https://mlsanity.com/paper/2310.05236/) (2023)
- [Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training](https://mlsanity.com/paper/2409.16767/) (2024)
- [Physics-informed, boundary-constrained Gaussian process regression for the reconstruction of fluid flow fields](https://mlsanity.com/paper/2507.17582/) (2025)
- [Likelihood Ratio Tests by Kernel Gaussian Embedding](https://mlsanity.com/paper/2508.07982/) (2025)
- [Asymptotically Optimal Change Detection for Unnormalized Pre- and Post-Change Distributions](https://mlsanity.com/paper/2410.14615/) (2024)

## Abstract (arXiv)

We consider the analysis of probability distributions through their associated covariance operators from reproducing kernel Hilbert spaces. We show that the von Neumann entropy and relative entropy of these operators are intimately related to the usual notions of Shannon entropy and relative entropy, and share many of their properties. They come together with efficient estimation algorithms from various oracles on the probability distributions. We also consider product spaces and show that for tensor product kernels, we can define notions of mutual information and joint entropies, which can then characterize independence perfectly, but only partially conditional independence. We finally show how these new notions of relative entropy lead to new upper-bounds on log partition functions, that can be used together with convex optimization within variational inference methods, providing a new family of probabilistic inference methods.