# Fast, close, non-singular and property-preserving approximations of entropic measures

arXiv: https://arxiv.org/abs/2505.14234 · HTML: https://mlsanity.com/paper/2505.14234/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 25 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 5 of 5)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes](https://mlsanity.com/paper/1805.08114/) (2018)
- [Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning](https://mlsanity.com/paper/1611.05128/) (2016)
- [On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means](https://mlsanity.com/paper/1904.04017/) (2019)
- [Robust learning of data anomalies with analytically-solvable entropic outlier sparsification](https://mlsanity.com/paper/2112.11768/) (2021)

## Built on by (most cited first, 1 of 1)

- [On a joint simultaneous learning of relevant feature subsets and subspaces in regression-like problems](https://mlsanity.com/paper/2607.28080/) (2026)

## Abstract (arXiv)

Entropic measures like Shannon entropy (SE), its quantum mechanical analogue von Neumann entropy, and Kullback-Leibler divergence (KL) are key components in many tools used in physics, information theory, machine learning (ML) and quantum computing. Besides of the significant amounts of SE and KL computations required in these fields, the singularity of their gradients near zero is one of the central mathematical reason inducing the high cost, frequently low robustness and slow convergence of computational tools that rely on these concepts. Here we propose the Fast Entropic Approximations (FEA) - non-singular rational approximations of SE and symmetrized KL, that preserve their main mathematical properties and achieve a mean absolute errors of around 10^-3 (10-20 times better than comparable state-of-the-art computational approximations). We show that FEA allows up to around 2 times faster computation of SE and up to 37 times faster computation of symmetrized KL: it requires only 5 to 7 elementary computational operations, as compared to the tens of elementary operations behind SE and KL evaluations based on approximate logarithm schemes with table look-ups, bitshifts, or series approximations. On a set of common benchmarks for the feature selection problem in machine learning, we show that the combined effect of fewer elementary operations, low approximation error, preservation of main mathematical properties, and non-singular gradients allows much faster training of significantly-better models. We demonstrate that FEA enables ML feature extraction that is three orders of magnitude faster, and better in quality then the very popular LASSO feature extraction.