# Quantifying Relevance in Learning and Inference

arXiv: https://arxiv.org/abs/2202.00339 · HTML: https://mlsanity.com/paper/2202.00339/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 141 references that are in the map.
- First posted on arXiv 1 Feb 2022.

## Builds on (most cited first, 27 of 27)

- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015)
- [Comparing Dynamics: Deep Neural Networks versus Glassy Systems](https://mlsanity.com/paper/1803.06969/) (2018)
- [Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes](https://mlsanity.com/paper/1605.06444/) (2016)
- [Inverse statistical problems: from the inverse Ising problem to data science](https://mlsanity.com/paper/1702.01522/) (2017)
- [Separation and Concentration in Deep Networks](https://mlsanity.com/paper/2012.10424/) (2020)
- [Thermodynamics of Restricted Boltzmann Machines and related learning dynamics](https://mlsanity.com/paper/1803.01960/) (2018)
- [Emergence of Compositional Representations in Restricted Boltzmann Machines](https://mlsanity.com/paper/1611.06759/) (2016)
- [Modelling the influence of data structure on learning in neural networks: the hidden manifold model](https://mlsanity.com/paper/1909.11500/) (2019)
- [Determination of the edge of criticality in echo state networks through Fisher information maximization](https://mlsanity.com/paper/1603.03685/) (2016)
- [What is important about the No Free Lunch theorems?](https://mlsanity.com/paper/2007.10928/) (2020)
- [Restricted Boltzmann Machines as Models of Interacting Variables](https://mlsanity.com/paper/2103.15917/) (2021)
- [Statistical mechanics of the US Supreme Court](https://mlsanity.com/paper/1306.5004/) (2013)
- [Learning with hidden variables](https://mlsanity.com/paper/1506.00354/) (2015)
- [Optimal encoding in stochastic latent-variable Models](https://mlsanity.com/paper/1802.10361/) (2018)
- [Minimum Description Length codes are critical](https://mlsanity.com/paper/1809.00652/) (2018)
- [Thermodynamic Machine Learning through Maximum Work Production](https://mlsanity.com/paper/2006.15416/) (2020)
- [Zipf's law and criticality in multivariate data without fine-tuning](https://mlsanity.com/paper/1310.0448/) (2013)
- [Statistical models of neural activity, criticality, and Zipf's law](https://mlsanity.com/paper/1812.09123/) (2018)
- [Bayesian Inference of Minimally Complex Models with Interactions of Arbitrary Order](https://mlsanity.com/paper/2008.00520/) (2020)
- [A random energy approach to deep learning](https://mlsanity.com/paper/2112.09420/) (2021)
- [Multiscale relevance and informative encoding in neuronal spike trains](https://mlsanity.com/paper/1802.10354/) (2018)
- [Counting the learnable functions of structured data](https://mlsanity.com/paper/1903.12021/) (2019)

## Built on by (most cited first, 5 of 5)

- [Detach-ROCKET: Sequential feature selection for time series classification with random convolutional kernels](https://mlsanity.com/paper/2309.14518/) (2023)
- [A simple probabilistic neural network for machine understanding](https://mlsanity.com/paper/2210.13179/) (2022)
- [Universal Scale Laws for Colors and Patterns in Imagery](https://mlsanity.com/paper/2406.08149/) (2024)
- [Absolute abstraction: a renormalisation group approach](https://mlsanity.com/paper/2407.01656/) (2024)
- [Evaluating Large Language Models in Code Generation: INFINITE Methodology for Defining the Inference Index](https://mlsanity.com/paper/2503.05852/) (2025)

## Abstract (arXiv)

Learning is a distinctive feature of intelligent behaviour. High-throughput experimental data and Big Data promise to open new windows on complex systems such as cells, the brain or our societies. Yet, the puzzling success of Artificial Intelligence and Machine Learning shows that we still have a poor conceptual understanding of learning. These applications push statistical inference into uncharted territories where data is high-dimensional and scarce, and prior information on "true" models is scant if not totally absent. Here we review recent progress on understanding learning, based on the notion of "relevance". The relevance, as we define it here, quantifies the amount of information that a dataset or the internal representation of a learning machine contains on the generative model of the data. This allows us to define maximally informative samples, on one hand, and optimal learning machines on the other. These are ideal limits of samples and of machines, that contain the maximal amount of information about the unknown generative process, at a given resolution (or level of compression). Both ideal limits exhibit critical features in the statistical sense: Maximally informative samples are characterised by a power-law frequency distribution (statistical criticality) and optimal learning machines by an anomalously large susceptibility. The trade-off between resolution (i.e. compression) and relevance distinguishes the regime of noisy representations from that of lossy compression. These are separated by a special point characterised by Zipf's law statistics. This identifies samples obeying Zipf's law as the most compressed loss-less representations that are optimal in the sense of maximal relevance. Criticality in optimal learning machines manifests in an exponential degeneracy of energy levels, that leads to unusual thermodynamic properties.