Characterising the Inductive Biases of Neural Networks on Boolean Data
Not on the map. Why?
Abstract (arXiv)
Deep neural networks are renowned for their ability to generalise well across diverse tasks, even when heavily overparameterized. Existing works offer only partial explanations (for example, the NTK-based task-model alignment explanation neglects feature learning). Here, we provide an end-to-end, analytically tractable case study that links a network's inductive prior, its training dynamics including feature learning, and its eventual generalisation. Specifically, we exploit the one-to-one correspondence between depth-2 discrete fully connected networks and disjunctive normal form (DNF) formulas by training on Boolean functions. Under a Monte Carlo learning algorithm, our model exhibits predictable training dynamics and the emergence of interpretable features. This framework allows us to trace, in detail, how inductive bias and feature formation drive generalisation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 45 of its 67 references that are in the map. First posted on arXiv 29 May 2025.
Builds on
45 of 67 references.
References · 45
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
- 2018On the Spectral Bias of Neural Networkscited by 1,341
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
- 2022In-context Learning and Induction Headscited by 964
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
- 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
33 more
- 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
- 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
- 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
- 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
- 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
- 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 135
- 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 132
- 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
- 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 70
- 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
- 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
- 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
- 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
- 2020Is SGD a Bayesian sampler? Well, almostcited by 50
- 2021What can linearized neural networks actually say about generalization?cited by 49
- 2024Identifying Semantic Induction Heads to Understand In-Context Learningcited by 47
- 2024Neural Redshift: Random Networks are not Random Functionscited by 43
- 2019Why ResNet Works? Residuals Generalizecited by 39
- 2020Generalization bounds for deep learningcited by 38
- 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
- 2020Predicting the outputs of finite deep neural networks trained with noisy gradientscited by 31
- 2023Deep neural networks have an inbuilt Occam's razorcited by 28
- 2019Neural networks are a priori biased towards Boolean functions with low entropycited by 24
- 2022On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernelscited by 16
- 2024Understanding Visual Feature Reliance through the Lens of Complexitycited by 15
- 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
- 2019Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspectivecited by 11
- 2022Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regimecited by 11
- 2020Rethinking Weight Decay For Efficient Neural Network Pruningcited by 10
- 2024Learning High-Degree Parities: The Crucial Role of the Initializationcited by 7
- 2025Uncovering Branch specialization in InceptionV1 using k sparse autoencoderscited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024The twin peaks of learning neural networks
- 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse
- 2025DNNs May Determine Major Properties of Their Outputs Early, with Timing Possibly Driven by Bias
- 2026Biased Generalization in Diffusion Models
- 2025Data Distributional Properties As Inductive Bias for Systematic Generalization
- 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models
- 2025When Do Neural Networks Learn World Models?
- 2025Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks