# Applications of Entropy in Data Analysis and Machine Learning: A Review arXiv: https://arxiv.org/abs/2503.02921 ยท HTML: https://mlsanity.com/paper/2503.02921/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 290 references that are in the map. - First posted on arXiv 4 Mar 2025. ## Builds on (most cited first, 30 of 31) - [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019) - [Cross-Entropy Loss Functions: Theoretical Analysis and Applications](https://mlsanity.com/paper/2304.07288/) (2023) - [Stop Regressing: Training Value Functions via Classification for Scalable Deep RL](https://mlsanity.com/paper/2403.03950/) (2024) - [Structural Entropy Guided Graph Hierarchical Pooling](https://mlsanity.com/paper/2206.13510/) (2022) - [Smooth Loss Functions for Deep Top-k Classification](https://mlsanity.com/paper/1802.07595/) (2018) - [An Alternative Cross Entropy Loss for Learning-to-Rank](https://mlsanity.com/paper/1911.09798/) (2019) - [Entanglement-guided architectures of machine learning by quantum tensor network](https://mlsanity.com/paper/1803.09111/) (2018) - [Graph Entropy Guided Node Embedding Dimension Selection for Graph Neural Networks](https://mlsanity.com/paper/2105.03178/) (2021) - [Fast Policy Learning for Linear Quadratic Control with Entropy Regularization](https://mlsanity.com/paper/2311.14168/) (2023) - [Multivariate Time Series Forecasting with Transfer Entropy Graph](https://mlsanity.com/paper/2005.01185/) (2020) - [Fast Incremental von Neumann Graph Entropy Computation: Theory, Algorithm, and Applications](https://mlsanity.com/paper/1805.11769/) (2018) - [Quantitative Performance Assessment of CNN Units via Topological Entropy Calculation](https://mlsanity.com/paper/2103.09716/) (2021) - [Transfer entropy-based feedback improves performance in artificial neural networks](https://mlsanity.com/paper/1706.04265/) (2017) - [On the mapping between Hopfield networks and Restricted Boltzmann Machines](https://mlsanity.com/paper/2101.11744/) (2021) - [Robust learning of data anomalies with analytically-solvable entropic outlier sparsification](https://mlsanity.com/paper/2112.11768/) (2021) - [Robust and Complex Approach of Pathological Speech Signal Analysis](https://mlsanity.com/paper/2203.09402/) (2022) - [Multivariate Dependence Beyond Shannon Information](https://mlsanity.com/paper/1609.01233/) (2016) - [Extreme Entropy Machines: Robust information theoretic classification](https://mlsanity.com/paper/1501.05279/) (2015) - [Semi-supervised cross-entropy clustering with information bottleneck constraint](https://mlsanity.com/paper/1705.01601/) (2017) - [Multiscale Fluctuation-based Dispersion Entropy and its Applications to Neurological Diseases](https://mlsanity.com/paper/1902.10825/) (2019) - [Mixture-based estimation of entropy](https://mlsanity.com/paper/2010.04058/) (2020) - [Examining convolutional feature extraction using Maximum Entropy (ME) and Signal-to-Noise Ratio (SNR) for image classification](https://mlsanity.com/paper/2105.04097/) (2021) - [A Quantitative Comparison between Shannon and Tsallis Havrda Charvat Entropies Applied to Cancer Outcome Prediction](https://mlsanity.com/paper/2203.11943/) (2022) - [Learning in Convolutional Neural Networks Accelerated by Transfer Entropy](https://mlsanity.com/paper/2404.02943/) (2024) - [MEMe: An Accurate Maximum Entropy Method for Efficient Approximations in Large-Scale Machine Learning](https://mlsanity.com/paper/1906.01101/) (2019) - [Deep learning using Havrda-Charvat entropy for classification of pulmonary endomicroscopy](https://mlsanity.com/paper/2104.05450/) (2021) - [Quantifying Unknown Quantum Entanglement via a Hybrid Quantum-Classical Machine Learning Framework](https://mlsanity.com/paper/2204.11500/) (2022) - [Soft-Output Signal Detection for Cetacean Vocalizations Using Spectral Entropy, K-Means Clustering and the Continuous Wavelet Transform](https://mlsanity.com/paper/2211.01065/) (2022) - [Differentiating patients with obstructive sleep apnea from healthy controls based on heart rate - blood pressure coupling quantified by entropy-based indices](https://mlsanity.com/paper/2311.10752/) (2023) - [Graph-Based Multivariate Multiscale Dispersion Entropy: Efficient Implementation and Applications to Real-World Network Data](https://mlsanity.com/paper/2405.00518/) (2024) ## Built on by (most cited first, 3 of 3) - [Neural Tangent Kernel Analysis to Probe Convergence in Physics-informed Neural Solvers: PIKANs vs. PINNs](https://mlsanity.com/paper/2506.07958/) (2025) - [Towards Self-Robust LLMs: Intrinsic Prompt Noise Resistance via CoIPO](https://mlsanity.com/paper/2603.03314/) (2026) - [Perspectives on Tsallis Statistics for Artificial Intelligence](https://mlsanity.com/paper/2608.01223/) (2026) ## Abstract (arXiv) Since its origin in the thermodynamics of the 19th century, the concept of entropy has also permeated other fields of physics and mathematics, such as Classical and Quantum Statistical Mechanics, Information Theory, Probability Theory, Ergodic Theory and the Theory of Dynamical Systems. Specifically, we are referring to the classical entropies: the Boltzmann-Gibbs, von Neumann, Shannon, Kolmogorov-Sinai and topological entropies. In addition to their common name, which is historically justified (as we briefly describe in this review), other commonality of the classical entropies is the important role that they have played and are still playing in the theory and applications of their respective fields and beyond. Therefore, it is not surprising that, in the course of time, many other instances of the overarching concept of entropy have been proposed, most of them tailored to specific purposes. Following the current usage, we will refer to all of them, whether classical or new, simply as entropies. Precisely, the subject of this review is their applications in data analysis and machine learning. The reason for these particular applications is that entropies are very well suited to characterize probability mass distributions, typically generated by finite-state processes or symbolized signals. Therefore, we will focus on entropies defined as positive functionals on probability mass distributions and provide an axiomatic characterization that goes back to Shannon and Khinchin. Given the plethora of entropies in the literature, we have selected a representative group, including the classical ones. The applications summarized in this review finely illustrate the power and versatility of entropy in data analysis and machine learning.