# Deep Sequential Neural Network

arXiv: https://arxiv.org/abs/1410.0510 · HTML: https://mlsanity.com/paper/1410.0510/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2014.

## Built on by (most cited first, 7 of 7)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Scaling Diffusion Transformers to 16 Billion Parameters](https://mlsanity.com/paper/2407.11633/) (2024)
- [Unraveling the Mystery of Scaling Laws: Part I](https://mlsanity.com/paper/2403.06563/) (2024)
- [Spatial Mixture-of-Experts](https://mlsanity.com/paper/2211.13491/) (2022)
- [Computing a human-like reaction time metric from stable recurrent vision models](https://mlsanity.com/paper/2306.11582/) (2023)
- [The AI Mechanic: Acoustic Vehicle Characterization Neural Networks](https://mlsanity.com/paper/2205.09667/) (2022)

## Abstract (arXiv)

Neural Networks sequentially build high-level features through their successive layers. We propose here a new neural network model where each layer is associated with a set of candidate mappings. When an input is processed, at each layer, one mapping among these candidates is selected according to a sequential decision process. The resulting model is structured according to a DAG like architecture, so that a path from the root to a leaf node defines a sequence of transformations. Instead of considering global transformations, like in classical multilayer networks, this model allows us for learning a set of local transformations. It is thus able to process data with different characteristics through specific sequences of such local transformations, increasing the expression power of this model w.r.t a classical multilayered network. The learning algorithm is inspired from policy gradient techniques coming from the reinforcement learning domain and is used here instead of the classical back-propagation based gradient descent techniques. Experiments on different datasets show the relevance of this approach.