# Machines of finite depth: towards a formalization of neural networks

arXiv: https://arxiv.org/abs/2204.12786 · HTML: https://mlsanity.com/paper/2204.12786/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 27 Apr 2022.

## Builds on (most cited first, 18 of 18)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Universal Differential Equations for Scientific Machine Learning](https://mlsanity.com/paper/2001.04385/) (2020)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Monotone operator equilibrium networks](https://mlsanity.com/paper/2006.08591/) (2020)
- [A Differentiable Programming System to Bridge Machine Learning and Scientific Computing](https://mlsanity.com/paper/1907.07587/) (2019)
- [Fashionable Modelling with Flux](https://mlsanity.com/paper/1811.01457/) (2018)
- [Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients](https://mlsanity.com/paper/2010.01709/) (2020)
- [Gradient Regularization Improves Accuracy of Discriminative Models](https://mlsanity.com/paper/1712.09936/) (2017)
- [Towards a topological-geometrical theory of group equivariant non-expansive operators for data analysis and machine learning](https://mlsanity.com/paper/1812.11832/) (2018)
- [Automatic differentiation in ML: Where we are and where we should be going](https://mlsanity.com/paper/1810.11530/) (2018)
- [Effective Extensible Programming: Unleashing Julia on GPUs](https://mlsanity.com/paper/1712.03112/) (2017)
- [Joint inference and input optimization in equilibrium networks](https://mlsanity.com/paper/2111.13236/) (2021)
- [Getting to the Point. Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming](https://mlsanity.com/paper/2104.05372/) (2021)
- [Swift for TensorFlow: A portable, flexible platform for deep learning](https://mlsanity.com/paper/2102.13243/) (2021)
- [A Benchmark of Selected Algorithmic Differentiation Tools on Some Problems in Computer Vision and Machine Learning](https://mlsanity.com/paper/1807.10129/) (2018)
- [Parametric machines: a fresh approach to architecture search](https://mlsanity.com/paper/2007.02777/) (2020)

## Built on by (most cited first, 2 of 2)

- [Generalized Permutants and Graph GENEOs](https://mlsanity.com/paper/2206.14798/) (2022)
- [Neural network layers as parametric spans](https://mlsanity.com/paper/2208.00809/) (2022)

## Abstract (arXiv)

We provide a unifying framework where artificial neural networks and their architectures can be formally described as particular cases of a general mathematical construction--machines of finite depth. Unlike neural networks, machines have a precise definition, from which several properties follow naturally. Machines of finite depth are modular (they can be combined), efficiently computable and differentiable. The backward pass of a machine is again a machine and can be computed without overhead using the same procedure as the forward pass. We prove this statement theoretically and practically, via a unified implementation that generalizes several classical architectures--dense, convolutional, and recurrent neural networks with a rich shortcut structure--and their respective backpropagation rules.