Understanding Nonlinear Implicit Bias via Region Counts in Input Space
Not on the map. Why?
Abstract (arXiv)
One explanation for the strong generalization ability of neural networks is implicit bias. Yet, the definition and mechanism of implicit bias in non-linear contexts remains little understood. In this work, we propose to characterize implicit bias by the count of connected regions in the input space with the same predicted label. Compared with parameter-dependent metrics (e.g., norm or normalized margin), region count can be better adapted to nonlinear, overparameterized models, because it is determined by the function mapping and is invariant to reparametrization. Empirically, we found that small region counts align with geometrically simple decision boundaries and correlate well with good generalization performance. We also observe that good hyper-parameter choices such as larger learning rates and smaller batch sizes can induce small region counts. We further establish the theoretical connections and explain how larger learning rate can induce small region counts in neural networks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 58 of its 65 references that are in the map. First posted on arXiv 16 May 2025.
Builds on
58 of 65 references.
References · 58
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2017Squeeze-and-Excitation Networkscited by 4,789
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
- 2020Designing Network Design Spacescited by 960
- 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
38 more
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2019Fantastic Generalization Measures and Where to Find Themcited by 513
- 2017Deep Layer Aggregationcited by 436
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
- 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 279
- 2017Three Factors Influencing Minima in SGDcited by 273
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
- 2018Towards Fast Computation of Certified Robustness for ReLU Networkscited by 255
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
- 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 189
- 2018Gradient descent aligns the layers of deep linear networkscited by 182
- 2020Directional convergence and alignment in deep learningcited by 178
- 2019Complexity of Linear Regions in Deep Networkscited by 167
- 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
- 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
- 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 161
- 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
- 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
- 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 114
- 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
- 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
- 2018Risk and parameter convergence of logistic regressioncited by 85
- 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
- 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
- 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
- 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
- 2019Conservative set valued fields, automatic differentiation, stochastic gradient method and deep learningcited by 72
- 2018Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Ratecited by 64
- 2022Can Neural Nets Learn the Same Model Twice? Investigating Reproducibility and Double Descent from the Decision Boundary Perspectivecited by 57
- 2022Learning threshold neurons via the "edge of stability"cited by 51
- 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
50 most cited of 58 All 58 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.