# Smooth Mathematical Function from Compact Neural Networks

arXiv: https://arxiv.org/abs/2301.00181 · HTML: https://mlsanity.com/paper/2301.00181/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 31 Dec 2022.

## Builds on (most cited first, 12 of 12)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [Sobolev Training for Neural Networks](https://mlsanity.com/paper/1706.04859/) (2017)
- [Activation Functions: Comparison of trends in Practice and Research for Deep Learning](https://mlsanity.com/paper/1811.03378/) (2018)
- [On weight initialization in deep neural networks](https://mlsanity.com/paper/1704.08863/) (2017)
- [Meta-Learning Requires Meta-Augmentation](https://mlsanity.com/paper/2007.05549/) (2020)
- [A Comprehensive Analysis of Deep Regression](https://mlsanity.com/paper/1803.08450/) (2018)
- [TRADI: Tracking deep neural network weight distributions for uncertainty estimation](https://mlsanity.com/paper/1912.11316/) (2019)
- [Approximating smooth functions by deep neural networks with sigmoid activation function](https://mlsanity.com/paper/2010.04596/) (2020)
- [Meta-learning autoencoders for few-shot prediction](https://mlsanity.com/paper/1807.09912/) (2018)

## Abstract (arXiv)

This is paper for the smooth function approximation by neural networks (NN). Mathematical or physical functions can be replaced by NN models through regression. In this study, we get NNs that generate highly accurate and highly smooth function, which only comprised of a few weight parameters, through discussing a few topics about regression. First, we reinterpret inside of NNs for regression; consequently, we propose a new activation function--integrated sigmoid linear unit (ISLU). Then special charateristics of metadata for regression, which is different from other data like image or sound, is discussed for improving the performance of neural networks. Finally, the one of a simple hierarchical NN that generate models substituting mathematical function is presented, and the new batch concept ``meta-batch" which improves the performance of NN several times more is introduced. The new activation function, meta-batch method, features of numerical data, meta-augmentation with metaparameters, and a structure of NN generating a compact multi-layer perceptron(MLP) are essential in this study.