# Estimating R'enyi's α-Cross-Entropies in a Matrix-Based Way

arXiv: https://arxiv.org/abs/2109.11737 · HTML: https://mlsanity.com/paper/2109.11737/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 77 references that are in the map.
- First posted on arXiv 24 Sep 2021.

## Builds on (most cited first, 8 of 8)

- [R'enyi Divergence Variational Inference](https://mlsanity.com/paper/1602.02311/) (2016)
- [Characteristic and Universal Tensor Product Kernels](https://mlsanity.com/paper/1708.08157/) (2017)
- [A Wild Bootstrap for Degenerate Kernel Tests](https://mlsanity.com/paper/1408.5404/) (2014)
- [A Kernel Independence Test for Random Processes](https://mlsanity.com/paper/1402.4501/) (2014)
- [Convergence guarantees for kernel-based quadrature rules in misspecified settings](https://mlsanity.com/paper/1605.07254/) (2016)
- [A Kernel Test for Three-Variable Interactions](https://mlsanity.com/paper/1306.2281/) (2013)
- [Information Theoretic Learning with Infinitely Divisible Kernels](https://mlsanity.com/paper/1301.3551/) (2013)
- [Characteristic Kernels and Infinitely Divisible Distributions](https://mlsanity.com/paper/1403.7304/) (2014)

## Built on by (most cited first, 1 of 1)

- [Kullback-Leibler and Renyi divergences in reproducing kernel Hilbert space and Gaussian process settings](https://mlsanity.com/paper/2207.08406/) (2022)

## Abstract (arXiv)

Conventional information-theoretic quantities assume access to probability distributions. Estimating such distributions is not trivial. Here, we consider function-based formulations of cross entropy that sidesteps this a priori estimation requirement. We propose three measures of R'enyi's α-cross-entropies in the setting of reproducing-kernel Hilbert spaces. Each measure has its appeals. We prove that we can estimate these measures in an unbiased, non-parametric, and minimax-optimal way. We do this via sample-constructed Gram matrices. This yields matrix-based estimators of R'enyi's α-cross-entropies. These estimators satisfy all of the axioms that R'enyi established for divergences. Our cross-entropies can thus be used for assessing distributional differences. They are also appropriate for handling high-dimensional distributions, since the convergence rate of our estimator is independent of the sample dimensionality. Python code for implementing these measures can be found at https://github.com/isledge/MBRCE