# Should Under-parameterized Student Networks Copy or Average Teacher Weights? arXiv: https://arxiv.org/abs/2311.01644 ยท HTML: https://mlsanity.com/paper/2311.01644/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 45 references that are in the map. - First posted on arXiv 3 Nov 2023. ## Builds on (most cited first, 29 of 29) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021) - [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020) - [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022) - [Towards moderate overparameterization: global convergence guarantees for training shallow neural networks](https://mlsanity.com/paper/1902.04674/) (2019) - [Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity](https://mlsanity.com/paper/1602.05897/) (2016) - [Theoretical insights into the optimization landscape of over-parameterized shallow neural networks](https://mlsanity.com/paper/1707.04926/) (2017) - [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021) - [Online stochastic gradient descent on non-convex losses from high-dimensional inference](https://mlsanity.com/paper/2003.10409/) (2020) - [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019) - [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017) - [Learning Single-Index Models with Shallow Neural Networks](https://mlsanity.com/paper/2210.15651/) (2022) - [The Landscape of Empirical Risk for Non-convex Losses](https://mlsanity.com/paper/1607.06534/) (2016) - [High-dimensional limit theorems for SGD: Effective dynamics and critical scaling](https://mlsanity.com/paper/2206.04030/) (2022) - [Neural Networks Efficiently Learn Low-Dimensional Representations with SGD](https://mlsanity.com/paper/2209.14863/) (2022) - [Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models](https://mlsanity.com/paper/2305.10633/) (2023) - [Learning time-scales in two-layers neural networks](https://mlsanity.com/paper/2303.00055/) (2023) - [On Learning Gaussian Multi-index Models with Gradient Flow](https://mlsanity.com/paper/2310.19793/) (2023) - [An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis](https://mlsanity.com/paper/1703.00560/) (2017) - [Learning a Single Neuron with Gradient Methods](https://mlsanity.com/paper/2001.05205/) (2020) - [Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks](https://mlsanity.com/paper/2202.00293/) (2022) - [The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks](https://mlsanity.com/paper/2006.01005/) (2020) - [Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron](https://mlsanity.com/paper/2302.10034/) (2023) - [Learning a Single Neuron with Bias Using Gradient Descent](https://mlsanity.com/paper/2106.01101/) (2021) - [On Single Index Models beyond Gaussian Data](https://mlsanity.com/paper/2307.15804/) (2023) - [Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II](https://mlsanity.com/paper/2107.10370/) (2021) - [Learning a Single Neuron for Non-monotonic Activation Functions](https://mlsanity.com/paper/2202.08064/) (2022) - [MLPGradientFlow: going with the flow of multilayer perceptrons (and finding minima fast and accurately)](https://mlsanity.com/paper/2301.10638/) (2023) - [Critical Point-Finding Methods Reveal Gradient-Flat Regions of Deep Network Losses](https://mlsanity.com/paper/2003.10397/) (2020) ## Built on by (most cited first, 8 of 8) - [Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence](https://mlsanity.com/paper/2411.08798/) (2024) - [Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding](https://mlsanity.com/paper/2402.05626/) (2024) - [Understanding Learning Invariance in Deep Linear Networks](https://mlsanity.com/paper/2506.13714/) (2025) - [Symmetry & Critical Points](https://mlsanity.com/paper/2408.14445/) (2024) - [Uncovering Critical Sets of Deep Neural Networks via Sample-Independent Critical Lifting](https://mlsanity.com/paper/2505.13582/) (2025) - [Data Augmentation Techniques to Reverse-Engineer Neural Network Weights from Input-Output Queries](https://mlsanity.com/paper/2511.20312/) (2025) - [Teacher Geometry Shapes Learnability in Teacher-Student Networks](https://mlsanity.com/paper/2609.09595/) (2026) - [Two-Timescale Fine-tuning Provably Learns New Features for Two-Layer ReLU Networks](https://mlsanity.com/paper/2609.34667/) (2026) ## Abstract (arXiv) Any continuous function f^* can be approximated arbitrarily well by a neural network with sufficiently many neurons k. We consider the case when f^* itself is a neural network with one hidden layer and k neurons. Approximating f^* with a neural network with n< k neurons can thus be seen as fitting an under-parameterized "student" network with n neurons to a "teacher" network with k neurons. As the student has fewer neurons than the teacher, it is unclear, whether each of the n student neurons should copy one of the teacher neurons or rather average a group of teacher neurons. For shallow neural networks with erf activation function and for the standard Gaussian input distribution, we prove that "copy-average" configurations are critical points if the teacher's incoming vectors are orthonormal and its outgoing weights are unitary. Moreover, the optimum among such configurations is reached when n-1 student neurons each copy one teacher neuron and the n-th student neuron averages the remaining k-n+1 teacher neurons. For the student network with n=1 neuron, we provide additionally a closed-form solution of the non-trivial critical point(s) for commonly used activation functions through solving an equivalent constrained optimization problem. Empirically, we find for the erf activation function that gradient flow converges either to the optimal copy-average critical point or to another point where each student neuron approximately copies a different teacher neuron. Finally, we find similar results for the ReLU activation function, suggesting that the optimal solution of underparameterized networks has a universal structure.