# Coding schemes in neural networks learning classification tasks

arXiv: https://arxiv.org/abs/2406.16689 · HTML: https://mlsanity.com/paper/2406.16689/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 79 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 30 of 36)

- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport](https://mlsanity.com/paper/1805.09545/) (2018)
- [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020)
- [A Conceptual Introduction to Hamiltonian Monte Carlo](https://mlsanity.com/paper/1701.02434/) (2017)
- [The Principles of Deep Learning Theory](https://mlsanity.com/paper/2106.10165/) (2021)
- [Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes](https://mlsanity.com/paper/1910.12478/) (2019)
- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [Mean Field Analysis of Neural Networks: A Central Limit Theorem](https://mlsanity.com/paper/1808.09372/) (2018)
- [Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks](https://mlsanity.com/paper/2205.09653/) (2022)
- [On the Role of Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2112.15121/) (2021)
- [Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics](https://mlsanity.com/paper/2012.04728/) (2020)
- [Non-Gaussian processes and neural networks at finite widths](https://mlsanity.com/paper/1910.00019/) (2019)
- [Asymptotics of Wide Networks from Feynman Diagrams](https://mlsanity.com/paper/1909.11304/) (2019)
- [Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization](https://mlsanity.com/paper/2012.04030/) (2020)
- [Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape](https://mlsanity.com/paper/1907.02911/) (2019)
- [Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs](https://mlsanity.com/paper/2112.15383/) (2021)
- [Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks](https://mlsanity.com/paper/2105.02716/) (2021)
- [Grokking as a First Order Phase Transition in Two Layer Networks](https://mlsanity.com/paper/2310.03789/) (2023)
- [Bayes-optimal Learning of Deep Random Networks of Extensive-width](https://mlsanity.com/paper/2302.00375/) (2023)
- [Scaling MLPs: A Tale of Inductive Bias](https://mlsanity.com/paper/2306.13575/) (2023)

## Built on by (most cited first, 11 of 11)

- [Adaptive kernel predictors from feature-learning infinite limits of neural networks](https://mlsanity.com/paper/2502.07998/) (2025)
- [High-dimensional learning of narrow neural networks](https://mlsanity.com/paper/2409.13904/) (2024)
- [From Kernels to Features: A Multi-Scale Adaptive Theory of Feature Learning](https://mlsanity.com/paper/2502.03210/) (2025)
- [Statistical mechanics of extensive-width Bayesian neural networks near interpolation](https://mlsanity.com/paper/2505.24849/) (2025)
- [A mathematical theory for understanding when abstract representations emerge in neural networks](https://mlsanity.com/paper/2510.09816/) (2025)
- [Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexity](https://mlsanity.com/paper/2512.04165/) (2025)
- [A unified theory of feature learning in RNNs and DNNs](https://mlsanity.com/paper/2602.15593/) (2026)
- [Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation](https://mlsanity.com/paper/2501.18530/) (2025)
- [Kernel Renormalization in Bayesian Deep Neural Networks: the Equivalent Wishart Ansatz in the Proportional Regime](https://mlsanity.com/paper/2605.29684/) (2026)
- [A Study of Hybrid and Evolutionary Metaheuristics for Single Hidden Layer Feedforward Neural Network Architecture](https://mlsanity.com/paper/2506.15737/) (2025)
- [Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation](https://mlsanity.com/paper/2510.24616/) (2025)

## Abstract (arXiv)

Neural networks posses the crucial ability to generate meaningful representations of task-dependent features. Indeed, with appropriate scaling, supervised learning in neural networks can result in strong, task-dependent feature learning. However, the nature of the emergent representations, which we call the `coding scheme', is still unclear. To understand the emergent coding scheme, we investigate fully-connected, wide neural networks learning classification tasks using the Bayesian framework where learning shapes the posterior distribution of the network weights. Consistent with previous findings, our analysis of the feature learning regime (also known as `non-lazy', `rich', or `mean-field' regime) shows that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity. In linear networks, an analog coding scheme of the task emerges. Despite the strong representations, the mean predictor is identical to the lazy case. In nonlinear networks, spontaneous symmetry breaking leads to either redundant or sparse coding schemes. Our findings highlight how network properties such as scaling of weights and neuronal nonlinearity can profoundly influence the emergent representations.