# Non-Autoregressive vs Autoregressive Neural Networks for System Identification

arXiv: https://arxiv.org/abs/2105.02027 · HTML: https://mlsanity.com/paper/2105.02027/

## Key facts

- Cited by 1 papers in the map (data through 7 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 17 of 17)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Professor Forcing: A New Algorithm for Training Recurrent Networks](https://mlsanity.com/paper/1610.09038/) (2016)
- [Regularizing and Optimizing LSTM Language Models](https://mlsanity.com/paper/1708.02182/) (2017)
- [A System for Massively Parallel Hyperparameter Tuning](https://mlsanity.com/paper/1810.05934/) (2018)
- [fastai: A Layered API for Deep Learning](https://mlsanity.com/paper/2002.04688/) (2020)
- [Neural Lander: Stable Drone Landing Control using Learned Dynamics](https://mlsanity.com/paper/1811.08027/) (2018)
- [Unbiasing Truncated Backpropagation Through Time](https://mlsanity.com/paper/1705.08209/) (2017)
- [Deep State Space Models for Nonlinear System Identification](https://mlsanity.com/paper/2003.14162/) (2020)
- [Deep Convolutional Networks in System Identification](https://mlsanity.com/paper/1909.01730/) (2019)
- [Fast Wavenet Generation Algorithm](https://mlsanity.com/paper/1611.09482/) (2016)
- ["Parallel Training Considered Harmful?": Comparing series-parallel and parallel feedforward network training](https://mlsanity.com/paper/1706.07119/) (2017)
- [Neural Networks Versus Conventional Filters for Inertial-Sensor-based Attitude Estimation](https://mlsanity.com/paper/2005.06897/) (2020)

## Built on by (most cited first, 1 of 1)

- [Sparse Bayesian Deep Learning for Dynamic System Identification](https://mlsanity.com/paper/2107.12910/) (2021)

## Abstract (arXiv)

The application of neural networks to non-linear dynamic system identification tasks has a long history, which consists mostly of autoregressive approaches. Autoregression, the usage of the model outputs of previous time steps, is a method of transferring a system state between time steps, which is not necessary for modeling dynamic systems with modern neural network structures, such as gated recurrent units (GRUs) and Temporal Convolutional Networks (TCNs). We compare the accuracy and execution performance of autoregressive and non-autoregressive implementations of a GRU and TCN on the simulation task of three publicly available system identification benchmarks. Our results show, that the non-autoregressive neural networks are significantly faster and at least as accurate as their autoregressive counterparts. Comparisons with other state-of-the-art black-box system identification methods show, that our implementation of the non-autoregressive GRU is the best performing neural network-based system identification method, and in the benchmarks without extrapolation, the best performing black-box method.