# Nonlinear functional regression by functional deep neural network with kernel embedding

arXiv: https://arxiv.org/abs/2401.02890 · HTML: https://mlsanity.com/paper/2401.02890/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 85 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 29 of 29)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Multipole Graph Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2006.09535/) (2020)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [Model Reduction and Neural Networks for Parametric PDEs](https://mlsanity.com/paper/2005.03180/) (2020)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Optimal approximation of continuous functions by very deep ReLU networks](https://mlsanity.com/paper/1802.03620/) (2018)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions](https://mlsanity.com/paper/1502.06800/) (2015)
- [Deep Learning on Image Denoising: An overview](https://mlsanity.com/paper/1912.13171/) (2019)
- [Operator-valued Kernels for Learning from Functional Response Data](https://mlsanity.com/paper/1510.08231/) (2015)
- [Convergence Rates for Learning Linear Operators from Noisy Data](https://mlsanity.com/paper/2108.12515/) (2021)
- [Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces](https://mlsanity.com/paper/2201.00217/) (2022)
- [SwitchNet: a neural network model for forward and inverse scattering problems](https://mlsanity.com/paper/1810.09675/) (2018)
- [Theory of Deep Convolutional Neural Networks III: Approximating Radial Functions](https://mlsanity.com/paper/2107.00896/) (2021)
- [Deep Learning for Functional Data Analysis with Adaptive Basis Layers](https://mlsanity.com/paper/2106.10414/) (2021)
- [Learning to Synthesize: Robust Phase Retrieval at Low Photon counts](https://mlsanity.com/paper/1907.11713/) (2019)
- [Convergence guarantees for kernel-based quadrature rules in misspecified settings](https://mlsanity.com/paper/1605.07254/) (2016)
- [scikit-fda: A Python Package for Functional Data Analysis](https://mlsanity.com/paper/2211.02566/) (2022)
- [Deep Neural Networks for Rotation-Invariance Approximation and Learning](https://mlsanity.com/paper/1904.01814/) (2019)
- [Single Channel Audio Source Separation using Convolutional Denoising Autoencoders](https://mlsanity.com/paper/1703.08019/) (2017)
- [Local approximation of operators](https://mlsanity.com/paper/2202.06392/) (2022)
- [Approximation of Nonlinear Functionals Using Deep ReLU Networks](https://mlsanity.com/paper/2304.04443/) (2023)

## Built on by (most cited first, 7 of 7)

- [Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss](https://mlsanity.com/paper/2402.00152/) (2024)
- [Optimal Convergence Rates for Neural Operators](https://mlsanity.com/paper/2412.17518/) (2024)
- [Highly Adaptive Ridge](https://mlsanity.com/paper/2410.02680/) (2024)
- [Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity](https://mlsanity.com/paper/2605.08811/) (2026)
- [Sparse-Aware Neural Networks for Nonlinear Functionals: Mitigating the Exponential Dependence on Dimension](https://mlsanity.com/paper/2604.06774/) (2026)
- [FAME: Adaptive Functional Attention with Expert Routing for Function-on-Function Regression](https://mlsanity.com/paper/2510.00621/) (2025)
- [Neural Scaling Laws of Transformer Operator Network](https://mlsanity.com/paper/2609.33533/) (2026)

## Abstract (arXiv)

Recently, deep learning has been widely applied in functional data analysis (FDA) with notable empirical success. However, the infinite dimensionality of functional data necessitates an effective dimension reduction approach for functional learning tasks, particularly in nonlinear functional regression. In this paper, we introduce a functional deep neural network with an adaptive and discretization-invariant dimension reduction method. Our functional network architecture consists of three parts: first, a kernel embedding step that features an integral transformation with an adaptive smooth kernel; next, a projection step that utilizes eigenfunction bases based on a projection Mercer kernel for the dimension reduction; and finally, a deep ReLU neural network is employed for the prediction. Explicit rates of approximating nonlinear smooth functionals across various input function spaces by our proposed functional network are derived. Additionally, we conduct a generalization analysis for the empirical risk minimization (ERM) algorithm applied to our functional net, by employing a novel two-stage oracle inequality and the established functional approximation results. Ultimately, we conduct numerical experiments on both simulated and real datasets to demonstrate the effectiveness and benefits of our functional net.