mlsanityData to 7 Oct 2026

Information Theory with Kernel Methods

arXiv 2202.08545 ↗First posted 17 Feb 2022cs.IT cs.LG math.ITcited by 37

Not on the map. Why?

Abstract (arXiv)

We consider the analysis of probability distributions through their associated covariance operators from reproducing kernel Hilbert spaces. We show that the von Neumann entropy and relative entropy of these operators are intimately related to the usual notions of Shannon entropy and relative entropy, and share many of their properties. They come together with efficient estimation algorithms from various oracles on the probability distributions. We also consider product spaces and show that for tensor product kernels, we can define notions of mutual information and joint entropies, which can then characterize independence perfectly, but only partially conditional independence. We finally show how these new notions of relative entropy lead to new upper-bounds on log partition functions, that can be used together with convex optimization within variational inference methods, providing a new family of probabilistic inference methods.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 12 of its 72 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

12 of 72 references.

References · 12

  1. 2018Demystifying MMD GANscited by 1,268
  2. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  3. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 360
  4. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  5. 2016Generalization Properties of Learning with Random Featurescited by 163
  6. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
  7. 2015Less is More: Nystr"om Computational Regularizationcited by 89
  8. 2020Finding Global Minima via Kernel Approximationscited by 23
  9. 2021A Dimension-free Computational Upper-bound for Smooth Optimal Transport Estimationcited by 17
  10. 2017Optimal rates of entropy estimation over Lipschitz ballscited by 15
  11. 2018Relating Leverage Scores and Density using Regularized Christoffel Functionscited by 10
  12. 2021Infinite-Dimensional Sums-of-Squares for Optimal Controlcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 14; 2025: 8; 2026: 4.

37 papers.

Sort:

By area:

  1. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  2. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  3. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  4. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 121
  5. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Modelscited by 31
  6. 2023Matrix Information Theory for Self-Supervised Learningcited by 21
  7. 2023Information Flow in Self-Supervised Learningcited by 16
  8. 2023Convergence Rates for Non-Log-Concave Sampling and Log-Partition Estimationcited by 10
  9. 2023Understanding Grokking Through A Robustness Viewpointcited by 10
  10. 2023The Representation Jensen-Shannon Divergencecited by 9
  11. 2024Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoningcited by 9
  12. 2022Sum-of-Squares Relaxations for Information Theory and Variational Inferencecited by 6
25 more
  1. 2024The Information of Large Language Model Geometrycited by 5
  2. 2022Auditing Differential Privacy in High Dimensions with the Kernel Quantum R'enyi Divergencecited by 3
  3. 2024Unveiling the Dynamics of Information Interplay in Supervised Learningcited by 3
  4. 2024Learning to Embed Distributions via Maximum Kernel Entropycited by 3
  5. 2024Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergencecited by 3
  6. 2025Information Shapes Koopman Representationcited by 3
  7. 2026The Maximum von Neumann Entropy Principle: Theory and Applications in Machine Learningcited by 3
  8. 2023Distributed Gradient Descent for Functional Learningcited by 2
  9. 2024A kernel-based analysis of Laplacian Eigenmapscited by 2
  10. 2024Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learningcited by 2
  11. 2024Estimating the Spectral Moments of the Kernel Integral Operator from Finite Sample Matricescited by 2
  12. 2025Kernel Embeddings and the Separation of Measure Phenomenoncited by 2
  13. 2023Enhancing Kernel Flexibility via Learning Asymmetric Locally-Adaptive Kernelscited by 1
  14. 2024Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Trainingcited by 1
  15. 2025Physics-informed, boundary-constrained Gaussian process regression for the reconstruction of fluid flow fieldscited by 1
  16. 2025Likelihood Ratio Tests by Kernel Gaussian Embeddingcited by 1
  17. 2026Exposing Diversity Bias in Deep Generative Models: Statistical Origins and Correction of Diversity Errorcited by 1
  18. 2024Asymptotically Optimal Change Detection for Unnormalized Pre- and Post-Change Distributionscited by 0
  19. 2024A Trust-Region Method for Graphical Stein Variational Inferencecited by 0
  20. 2024Variational Inference on the Boolean Hypercube with the Quantum Entropycited by 0
  21. 2025Information-Theoretic Perspectives on Optimizerscited by 0
  22. 2025Kernel Quantile Embeddings and Associated Probability Metricscited by 0
  23. 2025Kernel Trace Distance: Quantum Statistical Metric between Measures through RKHS Density Operatorscited by 0
  24. 2026LiQSS: Post-Transformer Linear Quantum-Inspired State-Space Tensor Networks for Real-Time 6Gcited by 0
  25. 2026Quantum Maximum Likelihood Prediction via Hilbert Space Embeddingscited by 0