# Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks

arXiv: https://arxiv.org/abs/1905.10264 · HTML: https://mlsanity.com/paper/1905.10264/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 20 of 20)

- [Overview frequency principle/spectral bias in deep learning](https://mlsanity.com/paper/2201.07395/) (2022)
- [HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging](https://mlsanity.com/paper/2203.02149/) (2022)
- [No Time to Train: Empowering Non-Parametric Networks for Few-shot 3D Scene Segmentation](https://mlsanity.com/paper/2404.04050/) (2024)
- [Exploring The Effect of High-frequency Components in GANs Training](https://mlsanity.com/paper/2103.11093/) (2021)
- [Exploring the Low-Pass Filtering Behavior in Image Super-Resolution](https://mlsanity.com/paper/2405.07919/) (2024)
- [How You Start Matters for Generalization](https://mlsanity.com/paper/2206.08558/) (2022)
- [Frequency-adaptive Multi-scale Deep Neural Networks](https://mlsanity.com/paper/2410.00053/) (2024)
- [Frequency Principle in Deep Learning Beyond Gradient-descent-based Training](https://mlsanity.com/paper/2101.00747/) (2021)
- [An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network](https://mlsanity.com/paper/2105.11675/) (2021)
- [A rationale from frequency perspective for grokking in training neural network](https://mlsanity.com/paper/2405.17479/) (2024)
- [Data-informed Deep Optimization](https://mlsanity.com/paper/2107.08166/) (2021)
- [Spectral Regularization Allows Data-frugal Learning over Combinatorial Spaces](https://mlsanity.com/paper/2210.02604/) (2022)
- [FW-GAN: Frequency-Driven Handwriting Synthesis with Wave-Modulated MLP Generator](https://mlsanity.com/paper/2508.21040/) (2025)
- [FreIE: Low-Frequency Spectral Bias in Neural Networks for Time-Series Tasks](https://mlsanity.com/paper/2510.25800/) (2025)
- [Spectral Gating Networks](https://mlsanity.com/paper/2602.07679/) (2026)
- [The Spectral Bias of Shallow Neural Network Learning is Shaped by the Choice of Non-linearity](https://mlsanity.com/paper/2503.10587/) (2025)
- [Frequency-adaptive tensor neural networks for high-dimensional multi-scale problems](https://mlsanity.com/paper/2508.15198/) (2025)
- [Noise Stability of Transformer Models](https://mlsanity.com/paper/2602.08287/) (2026)
- [Pseudo-differential-enhanced physics-informed neural networks](https://mlsanity.com/paper/2602.14663/) (2026)
- [SpiS-GAN: Spiral-Modulated Handwriting Synthesis with Star Operation](https://mlsanity.com/paper/2607.06949/) (2026)

## Abstract (arXiv)

It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well. An attempt of understanding this puzzle is to discover implicit biases underlying the training process of DNNs, such as the Frequency Principle (F-Principle), i.e., DNNs often fit target functions from low to high frequencies. Inspired by the F-Principle, we propose an effective model of linear F-Principle (LFP) dynamics which accurately predicts the learning results of two-layer ReLU neural networks (NNs) of large widths. This LFP dynamics is rationalized by a linearized mean field residual dynamics of NNs. Importantly, the long-time limit solution of this LFP dynamics is equivalent to the solution of a constrained optimization problem explicitly minimizing an FP-norm, in which higher frequencies of feasible solutions are more heavily penalized. Using this optimization formulation, an a priori estimate of the generalization error bound is provided, revealing that a higher FP-norm of the target function increases the generalization error. Overall, by explicitizing the implicit bias of the F-Principle as an explicit penalty for two-layer NNs, our work makes a step towards a quantitative understanding of the learning and generalization of general DNNs.