# Why Shallow Networks Struggle to Approximate and Learn High Frequencies

arXiv: https://arxiv.org/abs/2306.17301 · HTML: https://mlsanity.com/paper/2306.17301/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2023.

## Built on by (most cited first, 11 of 11)

- [Neural Redshift: Random Networks are not Random Functions](https://mlsanity.com/paper/2403.02241/) (2024)
- [On the expressiveness and spectral bias of KANs](https://mlsanity.com/paper/2410.01803/) (2024)
- [Approximation Rates for Shallow ReLUk Neural Networks on Sobolev Spaces via the Radon Transform](https://mlsanity.com/paper/2408.10996/) (2024)
- [ReLUs Are Sufficient for Learning Implicit Neural Representations](https://mlsanity.com/paper/2406.02529/) (2024)
- [Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential](https://mlsanity.com/paper/2502.18959/) (2025)
- [Structured and Balanced Multi-Component and Multi-Layer Neural Networks](https://mlsanity.com/paper/2407.00765/) (2024)
- [Condition Numbers and Eigenvalue Spectra of Shallow Networks on Spheres](https://mlsanity.com/paper/2511.02625/) (2025)
- [Multigrade Neural Network Approximation](https://mlsanity.com/paper/2601.16884/) (2026)
- [AYLA: Architecting a loss landscape in shallow neural networks to accelerate feature recovery](https://mlsanity.com/paper/2504.01875/) (2025)
- [Solving and learning advective multiscale Darcian dynamics with the Neural Basis Method](https://mlsanity.com/paper/2602.17776/) (2026)
- [Singularity Formation: Synergy in Theoretical, Numerical and Machine Learning Approaches](https://mlsanity.com/paper/2604.16842/) (2026)

## Abstract (arXiv)

In this work, we present a comprehensive study combining mathematical and computational analysis to explain why a two-layer neural network struggles to handle high frequencies in both approximation and learning, especially when machine precision, numerical noise, and computational cost are significant factors in practice. Specifically, we investigate the following fundamental computational issues: (1) the minimal numerical error achievable under finite precision, (2) the computational cost required to attain a given accuracy, and (3) the stability of the method with respect to perturbations. The core of our analysis lies in the conditioning of the representation and its learning dynamics. Explicit answers to these questions are provided, along with supporting numerical evidence.