# Activation Functions Not To Active: A Plausible Theory on Interpreting Neural Networks

arXiv: https://arxiv.org/abs/2305.00663 · HTML: https://mlsanity.com/paper/2305.00663/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 5 references that are in the map.
- First posted on arXiv 1 May 2023.

## Builds on (most cited first, 2 of 2)

- [Polynomial Regression As an Alternative to Neural Nets](https://mlsanity.com/paper/1806.06850/) (2018)
- [Towards a mathematical framework to inform Neural Network modelling via Polynomial Regression](https://mlsanity.com/paper/2102.03865/) (2021)

## Built on by (most cited first, 2 of 2)

- [Privacy-Preserving 3-Layer Neural Network Training](https://mlsanity.com/paper/2308.09531/) (2023)
- [LFFR: Logistic Function For (single-output) Regression](https://mlsanity.com/paper/2407.09955/) (2024)

## Abstract (arXiv)

Researchers commonly believe that neural networks model a high-dimensional space but cannot give a clear definition of this space. What is this space? What is its dimension? And does it has finite dimensions? In this paper, we develop a plausible theory on interpreting neural networks in terms of the role of activation functions in neural networks and define a high-dimensional (more precisely, an infinite-dimensional) space that neural networks including deep-learning networks could create. We show that the activation function acts as a magnifying function that maps the low-dimensional linear space into an infinite-dimensional space, which can distinctly identify the polynomial approximation of any multivariate continuous function of the variable values being the same features of the given dataset. Given a dataset with each example of d features f1, f2, cdots, fd, we believe that neural networks model a special space with infinite dimensions, each of which is a monomial prodi1, i2, cdots, id f1i1 f2i2 cdots fdid for some non-negative integers i1, i2, cdots, id ∈ mathbbZ0+=0,1,2,3,ldots . We term such an infinite-dimensional space a textit Super Space (SS). We see such a dimension as the minimum information unit. Every neuron node previously through an activation layer in neural networks is a textit Super Plane (SP) , which is actually a polynomial of infinite degree. This textit Super Space  is something like a coordinate system, in which every multivalue function can be represented by a textit Super Plane . We also show that training NNs could at least be reduced to solving a system of nonlinear equations. %solve sets of nonlinear equations