# Deep Probabilistic Time Series Forecasting using Augmented Recurrent Input for Dynamic Systems

arXiv: https://arxiv.org/abs/2106.05848 · HTML: https://mlsanity.com/paper/2106.05848/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 57 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Gaussian Processes for Big Data](https://mlsanity.com/paper/1309.6835/) (2013)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [When Gaussian Process Meets Big Data: A Review of Scalable GPs](https://mlsanity.com/paper/1807.01065/) (2018)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Bridging the Gap between Training and Inference for Neural Machine Translation](https://mlsanity.com/paper/1906.02448/) (2019)
- [Probabilistic Forecasting with Temporal Convolutional Neural Network](https://mlsanity.com/paper/1906.04397/) (2019)
- [Learning Stochastic Recurrent Networks](https://mlsanity.com/paper/1411.7610/) (2014)
- [Variational Gaussian Process State-Space Models](https://mlsanity.com/paper/1406.4905/) (2014)
- [Generalized Robust Bayesian Committee Machine for Large-scale Gaussian Process Regression](https://mlsanity.com/paper/1806.00720/) (2018)
- [Deep State Space Models for Nonlinear System Identification](https://mlsanity.com/paper/2003.14162/) (2020)
- [Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2102.00397/) (2021)
- [Learning Scalable Deep Kernels with Recurrent Structure](https://mlsanity.com/paper/1610.08936/) (2016)
- [Overcoming Mean-Field Approximations in Recurrent Gaussian Process Models](https://mlsanity.com/paper/1906.05828/) (2019)
- [Neural Dynamics Discovery via Gaussian Process Recurrent Neural Networks](https://mlsanity.com/paper/1907.00650/) (2019)
- [Stanza: A Nonlinear State Space Model for Probabilistic Inference in Non-Stationary Time Series](https://mlsanity.com/paper/2006.06553/) (2020)

## Built on by (most cited first, 1 of 1)

- [Output-Dependent Gaussian Process State-Space Model](https://mlsanity.com/paper/2212.07608/) (2022)

## Abstract (arXiv)

The demand of probabilistic time series forecasting has been recently raised in various dynamic system scenarios, for example, system identification and prognostic and health management of machines. To this end, we combine the advances in both deep generative models and state space model (SSM) to come up with a novel, data-driven deep probabilistic sequence model. Specifically, we follow the popular encoder-decoder generative structure to build the recurrent neural networks (RNN) assisted variational sequence model on an augmented recurrent input space, which could induce rich stochastic sequence dependency. Besides, in order to alleviate the inconsistency issue of the posterior between training and predicting as well as improving the mining of dynamic patterns, we (i) propose using a lagged hybrid output as input for the posterior at next time step, which brings training and predicting into alignment; and (ii) further devise a generalized auto-regressive strategy that encodes all the historical dependencies for the posterior. Thereafter, we first investigate the methodological characteristics of the proposed deep probabilistic sequence model on toy cases, and then comprehensively demonstrate the superiority of our model against existing deep probabilistic SSM models through extensive numerical experiments on eight system identification benchmarks from various dynamic systems. Finally, we apply our sequence model to a real-world centrifugal compressor forecasting problem, and again verify its outstanding performance by quantifying the time series predictive distribution.