Neuron Block Dynamics for XOR Classification with Zero-Margin
Not on the map. Why?
Abstract (arXiv)
The ability of neural networks to learn useful features through stochastic gradient descent (SGD) is a cornerstone of their success. Most theoretical analyses focus on regression or on classification tasks with a positive margin, where worst-case gradient bounds suffice. In contrast, we study zero-margin nonlinear classification by analyzing the Gaussian XOR problem, where inputs are Gaussian and the XOR decision boundary determines labels. In this setting, a non-negligible fraction of data lies arbitrarily close to the boundary, breaking standard margin-based arguments. Building on Glasgow's (2024) analysis, we extend the study of training dynamics from discrete to Gaussian inputs and develop a framework for the dynamics of neuron blocks. We show that neurons cluster into four directions and that block-level signals evolve coherently, a phenomenon essential in the Gaussian setting where individual neuron signals vary significantly. Leveraging this block perspective, we analyze generalization without relying on margin assumptions, adopting an average-case view that distinguishes regions of reliable prediction from regions of persistent error. Numerical experiments confirm the predicted two-phase block dynamics and demonstrate their robustness beyond the Gaussian setting.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 51 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
27 of 51 references.
References · 27
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
- 2020Directional convergence and alignment in deep learningcited by 172
- 2019Linearized two-layers neural networks in high dimensioncited by 155
- 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
- 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
- 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
- 2017Failures of Gradient-Based Deep Learningcited by 79
- 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
- 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
15 more
- 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
- 2023Learning time-scales in two-layers neural networkscited by 57
- 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
- 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
- 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
- 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
- 2023Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Datacited by 32
- 2023Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Timecited by 20
- 2024Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalizationcited by 17
- 2023From Tempered to Benign Overfitting in ReLU Neural Networkscited by 16
- 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
- 2024Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinationscited by 13
- 2023Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptronscited by 12
- 2024Matching the Statistical Query Lower Bound for k-Sparse Parity Problems with Sign Stochastic Gradient Descentcited by 7
- 2024Learning High-Degree Parities: The Crucial Role of the Initializationcited by 7
Built on by
Not cited yet.