# Smooth function approximation by deep neural networks with general activation functions

arXiv: https://arxiv.org/abs/1906.06903 · HTML: https://mlsanity.com/paper/1906.06903/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2019.

## Built on by (most cited first, 30 of 30)

- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks](https://mlsanity.com/paper/2109.02832/) (2021)
- [On Deep Generative Models for Approximation and Estimation of Distributions on Manifolds](https://mlsanity.com/paper/2302.13183/) (2023)
- [A likelihood approach to nonparametric estimation of a singular distribution using deep generative models](https://mlsanity.com/paper/2105.04046/) (2021)
- [Excess risk bound for deep learning under weak dependence](https://mlsanity.com/paper/2302.07503/) (2023)
- [Approximation speed of quantized vs. unquantized ReLU neural networks and beyond](https://mlsanity.com/paper/2205.11874/) (2022)
- [Sparse-penalized deep neural networks estimator under weak dependence](https://mlsanity.com/paper/2303.01406/) (2023)
- [Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks](https://mlsanity.com/paper/2404.03761/) (2024)
- [Robust deep learning from weakly dependent data](https://mlsanity.com/paper/2405.05081/) (2024)
- [Orthogonal Representation Learning for Estimating Causal Quantities](https://mlsanity.com/paper/2502.04274/) (2025)
- [CovNet: Covariance Networks for Functional Data on Multidimensional Domains](https://mlsanity.com/paper/2104.05021/) (2021)
- [Neural Networks with A La Carte Selection of Activation Functions](https://mlsanity.com/paper/2206.12166/) (2022)
- [Intrinsic and extrinsic deep learning on manifolds](https://mlsanity.com/paper/2302.08606/) (2023)
- [A Likelihood Based Approach to Distribution Regression Using Conditional Deep Generative Models](https://mlsanity.com/paper/2410.02025/) (2024)
- [Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds](https://mlsanity.com/paper/2410.21702/) (2024)
- [Deep Generative Models: Complexity, Dimensionality, and Approximation](https://mlsanity.com/paper/2504.00820/) (2025)
- [On the Suitability of Neural Networks as Building Blocks for The Design of Efficient Learned Indexes](https://mlsanity.com/paper/2203.14777/) (2022)
- [Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality](https://mlsanity.com/paper/2406.08321/) (2024)
- [Analysis of Structured Deep Kernel Networks](https://mlsanity.com/paper/2105.07228/) (2021)
- [What Has Been Enhanced in my Knowledge-Enhanced Language Model?](https://mlsanity.com/paper/2202.00964/) (2022)
- [Deep learning for psi-weakly dependent processes](https://mlsanity.com/paper/2302.00333/) (2023)
- [Penalized deep neural networks estimator with general loss functions under weak dependence](https://mlsanity.com/paper/2305.06230/) (2023)
- [Learned Bayesian Cram'er-Rao Bound for Unknown Measurement Models Using Score Neural Networks](https://mlsanity.com/paper/2502.00724/) (2025)
- [A general framework for deep learning](https://mlsanity.com/paper/2512.23425/) (2025)
- [ReLU integral probability metric and its applications](https://mlsanity.com/paper/2504.18897/) (2025)
- [Integral Signatures of Activation Functions: A 9-Dimensional Taxonomy and Stability Theory for Deep Learning](https://mlsanity.com/paper/2510.08456/) (2025)
- [Deep regression learning from dependent observations with minimum error entropy principle](https://mlsanity.com/paper/2603.11138/) (2026)
- [A Composite Activation Function for Learning Stable Binary Representations](https://mlsanity.com/paper/2605.11558/) (2026)
- [Modeling Covariate Transition for Efficient Estimation of Longitudinal Treatment Effects in Randomized Experiments](https://mlsanity.com/paper/2605.31443/) (2026)
- [Adaptive deep nonparametric regression from dependent data under covariate shift](https://mlsanity.com/paper/2607.20309/) (2026)

## Abstract (arXiv)

There has been a growing interest in expressivity of deep neural networks. However, most of the existing work about this topic focuses only on the specific activation function such as ReLU or sigmoid. In this paper, we investigate the approximation ability of deep neural networks with a broad class of activation functions. This class of activation functions includes most of frequently used activation functions. We derive the required depth, width and sparsity of a deep neural network to approximate any H"older smooth function upto a given approximation error for the large class of activation functions. Based on our approximation error analysis, we derive the minimax optimality of the deep neural network estimators with the general activation functions in both regression and classification problems.