# Phase transitions in the mini-batch size for sparse and dense two-layer neural networks arXiv: https://arxiv.org/abs/2305.06435 ยท HTML: https://mlsanity.com/paper/2305.06435/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 26 of its 80 references that are in the map. - First posted on arXiv 10 May 2023. ## Builds on (most cited first, 26 of 26) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016) - [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017) - [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018) - [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015) - [Learning curves of generic features maps for realistic datasets with a teacher-student model](https://mlsanity.com/paper/2102.08127/) (2021) - [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019) - [Revisiting Small Batch Training for Deep Neural Networks](https://mlsanity.com/paper/1804.07612/) (2018) - [Learning time-scales in two-layers neural networks](https://mlsanity.com/paper/2303.00055/) (2023) - [Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks](https://mlsanity.com/paper/2202.00293/) (2022) - [Unveiling the structure of wide flat minima in neural networks](https://mlsanity.com/paper/2107.01163/) (2021) - [Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines](https://mlsanity.com/paper/2105.13889/) (2021) - [Entropic gradient descent algorithms and wide flat minima](https://mlsanity.com/paper/2006.07897/) (2020) - [Learning through atypical "phase transitions" in overparameterized neural networks](https://mlsanity.com/paper/2110.00683/) (2021) - [Information-theoretic thresholds for community detection in sparse networks](https://mlsanity.com/paper/1601.02658/) (2016) - [Learning curves for the multi-class teacher-student perceptron](https://mlsanity.com/paper/2203.12094/) (2022) - [The backtracking survey propagation algorithm for solving random K-SAT problems](https://mlsanity.com/paper/1508.05117/) (2015) - [Deep learning via message passing algorithms based on belief propagation](https://mlsanity.com/paper/2110.14583/) (2021) - [Learning from Survey Propagation: a Neural Network for MAX-E-3-SAT](https://mlsanity.com/paper/2012.06344/) (2020) - [Solving non-linear Kolmogorov equations in large dimensions by using deep learning: a numerical comparison of discretization schemes](https://mlsanity.com/paper/2012.07747/) (2020) - [Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks](https://mlsanity.com/paper/2201.00032/) (2021) - [Hard Optimization Problems have Soft Edges](https://mlsanity.com/paper/2209.04824/) (2022) - [Chaos and correlated avalanches in excitatory neural networks with synaptic plasticity](https://mlsanity.com/paper/1610.00252/) (2016) - [Optimal Mini-Batch Size Selection for Fast Gradient Descent](https://mlsanity.com/paper/1911.06459/) (2019) - [Recurrent neural networks that generalize from examples and optimize by dreaming](https://mlsanity.com/paper/2204.07954/) (2022) ## Built on by (most cited first, 9 of 9) - [Complex Recurrent Spectral Network](https://mlsanity.com/paper/2312.07296/) (2023) - [SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training](https://mlsanity.com/paper/2505.23489/) (2025) - [Parallel Learning by Multitasking Neural Networks](https://mlsanity.com/paper/2308.04106/) (2023) - [Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems](https://mlsanity.com/paper/2309.05337/) (2023) - [Fast Analysis of the OpenAI O1-Preview Model in Solving Random K-SAT Problem: Does the LLM Solve the Problem Itself or Call an External SAT Solver?](https://mlsanity.com/paper/2409.11232/) (2024) - [Spring-block theory of feature learning in deep neural networks](https://mlsanity.com/paper/2407.19353/) (2024) - [Unsupervised detection of semantic correlations in big data](https://mlsanity.com/paper/2411.02126/) (2024) - [Role of Delay in Brain Dynamics](https://mlsanity.com/paper/2410.11384/) (2024) - [Engineered Ordinary Differential Equations as Classification Algorithm (EODECA): thorough characterization and testing](https://mlsanity.com/paper/2312.14681/) (2023) ## Abstract (arXiv) The use of mini-batches of data in training artificial neural networks is nowadays very common. Despite its broad usage, theories explaining quantitatively how large or small the optimal mini-batch size should be are missing. This work presents a systematic attempt at understanding the role of the mini-batch size in training two-layer neural networks. Working in the teacher-student scenario, with a sparse teacher, and focusing on tasks of different complexity, we quantify the effects of changing the mini-batch size m. We find that often the generalization performances of the student strongly depend on m and may undergo sharp phase transitions at a critical value mc, such that for mmc the student learns perfectly or generalizes very well the teacher. Phase transitions are induced by collective phenomena firstly discovered in statistical mechanics and later observed in many fields of science. Observing a phase transition by varying the mini-batch size across different architectures raises several questions about the role of this hyperparameter in the neural network learning process.