# Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension

arXiv: https://arxiv.org/abs/2604.06774 · HTML: https://mlsanity.com/paper/2604.06774/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Optimal approximation of piecewise smooth functions using deep ReLU neural networks](https://mlsanity.com/paper/1709.05289/) (2017)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds](https://mlsanity.com/paper/1808.10038/) (2018)
- [Deep ReLU network approximation of functions on a manifold](https://mlsanity.com/paper/1908.00695/) (2019)
- [Provable approximation properties for deep neural networks](https://mlsanity.com/paper/1509.07385/) (2015)
- [Convolutional Neural Networks Analyzed via Convolutional Sparse Coding](https://mlsanity.com/paper/1607.08194/) (2016)
- [Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks](https://mlsanity.com/paper/2109.02832/) (2021)
- [Deep nets for local manifold learning](https://mlsanity.com/paper/1607.07110/) (2016)
- [Theory of Deep Convolutional Neural Networks II: Spherical Analysis](https://mlsanity.com/paper/2007.14285/) (2020)
- [Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions](https://mlsanity.com/paper/2107.00896/) (2021)
- [Deep Learning for Functional Data Analysis with Adaptive Basis Layers](https://mlsanity.com/paper/2106.10414/) (2021)
- [Generalization Error Guaranteed Auto-Encoder-Based Nonlinear Model Reduction for Operator Learning](https://mlsanity.com/paper/2401.10490/) (2024)
- [Approximation of Functionals by Neural Network without Curse of Dimensionality](https://mlsanity.com/paper/2205.14421/) (2022)
- [Approximation of Nonlinear Functionals Using Deep ReLU Networks](https://mlsanity.com/paper/2304.04443/) (2023)
- [Nonlinear functional regression by functional deep neural network with kernel embedding](https://mlsanity.com/paper/2401.02890/) (2024)
- [Approximation analysis of CNNs from a feature extraction view](https://mlsanity.com/paper/2210.09041/) (2022)
- [Neural Operators for Nonlinear Functionals on RKHS](https://mlsanity.com/paper/2403.12187/) (2024)
- [SignReLU neural network and its approximation ability](https://mlsanity.com/paper/2210.10264/) (2022)
- [Spherical Analysis of Learning Nonlinear Functionals](https://mlsanity.com/paper/2410.01047/) (2024)
- [Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks](https://mlsanity.com/paper/2408.05540/) (2024)
- [DeepONet for Solving Nonlinear Partial Differential Equations with Physics-Informed Training](https://mlsanity.com/paper/2410.04344/) (2024)

## Built on by (most cited first, 1 of 1)

- [Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spaces](https://mlsanity.com/paper/2606.17419/) (2026)

## Abstract (arXiv)

Deep neural networks have emerged as powerful tools for learning operators defined over infinite-dimensional function spaces. However, existing theories frequently encounter difficulties related to dimensionality and limited interpretability. This work investigates how sparsity can help address these challenges in functional learning, a central ingredient in operator learning. We propose a framework that employs convolutional architectures to extract sparse features from a finite number of samples, together with deep fully connected networks to effectively approximate nonlinear functionals. Using universal discretization methods, we show that sparse approximators enable stable recovery from discrete samples. In addition, both the deterministic and the random sampling schemes are sufficient for our analysis. These findings lead to improved approximation rates and reduced sample sizes in various function spaces, including those with fast frequency decay and mixed smoothness. They also provide new theoretical insights into how sparsity can alleviate the curse of dimensionality in functional learning.