# Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits

arXiv: https://arxiv.org/abs/2512.14338 · HTML: https://mlsanity.com/paper/2512.14338/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 58 references that are in the map.
- First posted on arXiv 16 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Hopfield Networks is All You Need](https://mlsanity.com/paper/2008.02217/) (2020)
- [Dense Associative Memory for Pattern Recognition](https://mlsanity.com/paper/1606.01164/) (2016)
- [A vector-contraction inequality for Rademacher complexities](https://mlsanity.com/paper/1605.00251/) (2016)
- [Convergence of Gradient Descent on Separable Data](https://mlsanity.com/paper/1803.01905/) (2018)
- [A Kernel Theory of Modern Data Augmentation](https://mlsanity.com/paper/1803.06084/) (2018)
- [Risk and parameter convergence of logistic regression](https://mlsanity.com/paper/1803.07300/) (2018)
- [Provably Strict Generalisation Benefit for Invariance in Kernel Methods](https://mlsanity.com/paper/2106.02346/) (2021)
- [Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks](https://mlsanity.com/paper/2312.08550/) (2023)
- [Optimization Dynamics of Equivariant and Augmented Neural Networks](https://mlsanity.com/paper/2303.13458/) (2023)
- [Emergent Equivariance in Deep Ensembles](https://mlsanity.com/paper/2403.03103/) (2024)
- [Storage and Learning phase transitions in the Random-Features Hopfield Model](https://mlsanity.com/paper/2303.16880/) (2023)
- [Random Features Hopfield Networks generalize retrieval to previously unseen examples](https://mlsanity.com/paper/2407.05658/) (2024)
- [A new method for parameter estimation in probabilistic models: Minimum probability flow](https://mlsanity.com/paper/2007.09240/) (2020)
- [New Insights on Learning Rules for Hopfield Networks: Memory and Objective Function Minimisation](https://mlsanity.com/paper/2010.01472/) (2020)
- [Robust exponential memory in Hopfield networks](https://mlsanity.com/paper/1411.4625/) (2014)
- [Equivariant Neural Tangent Kernels](https://mlsanity.com/paper/2406.06504/) (2024)

## Abstract (arXiv)

Many learning problems are organized by group symmetries. While invariance is often imposed through architectures or group averaging, we ask when it can emerge from training on a finite random subset of an orbit. We study this question in classical Hopfield networks, where strict memorization can be expressed as a linear margin problem. Reparameterizing minimization of energy flow (MEF) as an exponential loss connects gradient descent to the corresponding minimum-norm hard-margin memorizer. Our main result shows that, for independent uniform samples from any finite permutation orbit, the exact sample hard-margin support vector machine (HSVM) concentrates exponentially around the invariant full-orbit HSVM. Consequently, an orbit-size-independent polynomial number of samples suffices both for approximate parameter invariance and for simultaneous memorization of every orbit element; directional convergence transfers this conclusion asymptotically to MEF gradient descent. For graph-isomorphism orbits, we characterize the invariant parameters as a three-dimensional subspace and show that every such orbit is memorizable. For cliques of fixed linear density, additional symmetry sharpens the uniform memorization bound to O(v4log(1/δ)), exponentially smaller than the orbit size. Together with experiments across several learning rules, these results give a finite-sample account of how optimization bias can recover symmetry from partial group-structured data.