# Hybrid State Space-based Learning for Sequential Data Prediction with Joint Optimization

arXiv: https://arxiv.org/abs/2309.10553 · HTML: https://mlsanity.com/paper/2309.10553/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 55 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 8 of 8)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [META-DES: A Dynamic Ensemble Selection Framework using Meta-Learning](https://mlsanity.com/paper/1810.01270/) (2018)
- [Improving LSTM Neural Networks for Better Short-Term Wind Power Predictions](https://mlsanity.com/paper/1907.00489/) (2019)
- [A Deep Convolutional Neural Network Model for improving WRF Forecasts](https://mlsanity.com/paper/2008.06489/) (2020)
- [Online Training of LSTM Networks in Distributed Systems for Variable Length Data Sequences](https://mlsanity.com/paper/1710.08744/) (2017)
- [A Hybrid Framework for Sequential Data Prediction with End-to-End Optimization](https://mlsanity.com/paper/2203.13787/) (2022)

## Abstract (arXiv)

We investigate nonlinear prediction/regression in an online setting and introduce a hybrid model that effectively mitigates, via a joint mechanism through a state space formulation, the need for domain-specific feature engineering issues of conventional nonlinear prediction models and achieves an efficient mix of nonlinear and linear components. In particular, we use recursive structures to extract features from raw sequential sequences and a traditional linear time series model to deal with the intricacies of the sequential data, e.g., seasonality, trends. The state-of-the-art ensemble or hybrid models typically train the base models in a disjoint manner, which is not only time consuming but also sub-optimal due to the separation of modeling or independent training. In contrast, as the first time in the literature, we jointly optimize an enhanced recurrent neural network (LSTM) for automatic feature extraction from raw data and an ARMA-family time series model (SARIMAX) for effectively addressing peculiarities associated with time series data. We achieve this by introducing novel state space representations for the base models, which are then combined to provide a full state space representation of the hybrid or the ensemble. Hence, we are able to jointly optimize both models in a single pass via particle filtering, for which we also provide the update equations. The introduced architecture is generic so that one can use other recurrent architectures, e.g., GRUs, traditional time series-specific models, e.g., ETS or other optimization methods, e.g., EKF, UKF. Due to such novel combination and joint optimization, we demonstrate significant improvements in widely publicized real life competition datasets. We also openly share our code for further research and replicability of our results.