# Neural networks with redundant representation: detecting the undetectable

arXiv: https://arxiv.org/abs/1911.12689 · HTML: https://mlsanity.com/paper/1911.12689/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2019.

## Built on by (most cited first, 12 of 12)

- [The emergence of a concept in shallow neural networks](https://mlsanity.com/paper/2109.00454/) (2021)
- [Differentiable Physics: A Position Piece](https://mlsanity.com/paper/2109.07573/) (2021)
- [Dense Hebbian neural networks: a replica symmetric picture of unsupervised learning](https://mlsanity.com/paper/2211.14067/) (2022)
- [Dense Hebbian neural networks: a replica symmetric picture of supervised learning](https://mlsanity.com/paper/2212.00606/) (2022)
- [Dense Hopfield Networks in the Teacher-Student Setting](https://mlsanity.com/paper/2401.04191/) (2024)
- [Regularization, early-stopping and dreaming: a Hopfield-like setup to address generalization and overfitting](https://mlsanity.com/paper/2308.01421/) (2023)
- [Hebbian Learning from First Principles](https://mlsanity.com/paper/2401.07110/) (2024)
- [Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines](https://mlsanity.com/paper/2406.09924/) (2024)
- [Recurrent neural networks that generalize from examples and optimize by dreaming](https://mlsanity.com/paper/2204.07954/) (2022)
- [Find the Lady: Permutation and Re-Synchronization of Deep Neural Networks](https://mlsanity.com/paper/2312.14182/) (2023)
- [Unsupervised and Supervised learning by Dense Associative Memory under replica symmetry breaking](https://mlsanity.com/paper/2312.09638/) (2023)
- [Beyond Disorder: Unveiling Cooperativeness in Multidirectional Associative Memories](https://mlsanity.com/paper/2503.04454/) (2025)

## Abstract (arXiv)

We consider a three-layer Sejnowski machine and show that features learnt via contrastive divergence have a dual representation as patterns in a dense associative memory of order P=4. The latter is known to be able to Hebbian-store an amount of patterns scaling as NP-1, where N denotes the number of constituting binary neurons interacting P-wisely. We also prove that, by keeping the dense associative network far from the saturation regime (namely, allowing for a number of patterns scaling only linearly with N, while P>2) such a system is able to perform pattern recognition far below the standard signal-to-noise threshold. In particular, a network with P=4 is able to retrieve information whose intensity is O(1) even in the presence of a noise O(√N) in the large N limit. This striking skill stems from a redundancy representation of patterns -- which is afforded given the (relatively) low-load information storage -- and it contributes to explain the impressive abilities in pattern recognition exhibited by new-generation neural networks. The whole theory is developed rigorously, at the replica symmetric level of approximation, and corroborated by signal-to-noise analysis and Monte Carlo simulations.