# A Tutorial on Neural Networks and Gradient-free Training

arXiv: https://arxiv.org/abs/2211.17217 · HTML: https://mlsanity.com/paper/2211.17217/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 14 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 1 of 1)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)

## Built on by (most cited first, 1 of 1)

- [A Novel Neural Filter to Improve Accuracy of Neural Network Models of Dynamic Systems](https://mlsanity.com/paper/2409.13654/) (2024)

## Abstract (arXiv)

This paper presents a compact, matrix-based representation of neural networks in a self-contained tutorial fashion. Specifically, we develop neural networks as a composition of several vector-valued functions. Although neural networks are well-understood pictorially in terms of interconnected neurons, neural networks are mathematical nonlinear functions constructed by composing several vector-valued functions. Using basic results from linear algebra, we represent a neural network as an alternating sequence of linear maps and scalar nonlinear functions, also known as activation functions. The training of neural networks requires the minimization of a cost function, which in turn requires the computation of a gradient. Using basic multivariable calculus results, the cost gradient is also shown to be a function composed of a sequence of linear maps and nonlinear functions. In addition to the analytical gradient computation, we consider two gradient-free training methods and compare the three training methods in terms of convergence rate and prediction accuracy.