# Optimal input representation in neural systems at the edge of chaos

arXiv: https://arxiv.org/abs/2107.05709 · HTML: https://mlsanity.com/paper/2107.05709/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 59 references that are in the map.
- First posted on arXiv 12 Jul 2021.

## Builds on (most cited first, 11 of 11)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [A Tutorial on Principal Component Analysis](https://mlsanity.com/paper/1404.1100/) (2014)
- [Reservoir computing approaches for representation and classification of multivariate time series](https://mlsanity.com/paper/1803.07870/) (2018)
- [On 1/n neural representation and robustness](https://mlsanity.com/paper/2012.04729/) (2020)
- [Criticality in the brain: A synthesis of neurobiology, models and cognition](https://mlsanity.com/paper/1707.05952/) (2017)
- [Unveiling the role of plasticity rules in reservoir computing](https://mlsanity.com/paper/2101.05848/) (2021)
- [Chasing the Echo State Property](https://mlsanity.com/paper/1811.10892/) (2018)
- [Power-law statistics and universal scaling in the absence of criticality](https://mlsanity.com/paper/1503.08033/) (2015)
- [Distributions of covariances as a window into the operational regime of neuronal networks](https://mlsanity.com/paper/1605.04153/) (2016)
- [Landau-Ginzburg theory of cortex dynamics: Scale-free avalanches emerge at the edge of synchronization](https://mlsanity.com/paper/1801.10356/) (2018)
- [Neutral theory and scale-free neural dynamics](https://mlsanity.com/paper/1703.05079/) (2017)

## Built on by (most cited first, 2 of 2)

- [Dynamical stability and chaos in artificial neural network trajectories along training](https://mlsanity.com/paper/2404.05782/) (2024)
- [Over the Edge of Chaos? Excess Complexity as a Roadblock to Artificial General Intelligence](https://mlsanity.com/paper/2407.03652/) (2024)

## Abstract (arXiv)

Shedding light onto how biological systems represent, process and store information in noisy environments is a key and challenging goal. A stimulating, though controversial, hypothesis poses that operating in dynamical regimes near the edge of a phase transition, i.e. at criticality or the "edge of chaos", can provide information-processing living systems with important operational advantages, creating, e.g., an optimal trade-off between robustness and flexibility. Here, we elaborate on a recent theoretical result, which establishes that the spectrum of covariance matrices of neural networks representing complex inputs in a robust way needs to decay as a power-law of the rank, with an exponent close to unity, a result that has been indeed experimentally verified in neurons of the mouse visual cortex. Aimed at understanding and mimicking these results, we construct an artificial neural network and train it to classify images. Remarkably, we find that the best performance in such a task is obtained when the network operates near the critical point, at which the eigenspectrum of the covariance matrix follows the very same statistics as actual neurons do. Thus, we conclude that operating near criticality can also have -- besides the usually alleged virtues -- the advantage of allowing for flexible, robust and efficient input representations.