# A Hybrid Framework for Sequential Data Prediction with End-to-End Optimization

arXiv: https://arxiv.org/abs/2203.13787 · HTML: https://mlsanity.com/paper/2203.13787/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 55 references that are in the map.
- First posted on arXiv 25 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Adaptive Neural Trees](https://mlsanity.com/paper/1807.06699/) (2018)
- [The Tree Ensemble Layer: Differentiability meets Conditional Computation](https://mlsanity.com/paper/2002.07772/) (2020)
- [Forecasting: theory and practice](https://mlsanity.com/paper/2012.03854/) (2020)
- [Neural Random Forests](https://mlsanity.com/paper/1604.07143/) (2016)
- [Soft Gradient Boosting Machine](https://mlsanity.com/paper/2006.04059/) (2020)
- [Hybrid Gradient Boosting Trees and Neural Networks for Forecasting Operating Room Data](https://mlsanity.com/paper/1801.07384/) (2018)
- [Improving LSTM Neural Networks for Better Short-Term Wind Power Predictions](https://mlsanity.com/paper/1907.00489/) (2019)
- [Highly Efficient Hierarchical Online Nonlinear Regression Using Second Order Methods](https://mlsanity.com/paper/1701.05053/) (2017)

## Built on by (most cited first, 1 of 1)

- [Hybrid State Space-based Learning for Sequential Data Prediction with Joint Optimization](https://mlsanity.com/paper/2309.10553/) (2023)

## Abstract (arXiv)

We investigate nonlinear prediction in an online setting and introduce a hybrid model that effectively mitigates, via an end-to-end architecture, the need for hand-designed features and manual model selection issues of conventional nonlinear prediction/regression methods. In particular, we use recursive structures to extract features from sequential signals, while preserving the state information, i.e., the history, and boosted decision trees to produce the final output. The connection is in an end-to-end fashion and we jointly optimize the whole architecture using stochastic gradient descent, for which we also provide the backward pass update equations. In particular, we employ a recurrent neural network (LSTM) for adaptive feature extraction from sequential data and a gradient boosting machinery (soft GBDT) for effective supervised regression. Our framework is generic so that one can use other deep learning architectures for feature extraction (such as RNNs and GRUs) and machine learning algorithms for decision making as long as they are differentiable. We demonstrate the learning behavior of our algorithm on synthetic data and the significant performance improvements over the conventional methods over various real life datasets. Furthermore, we openly share the source code of the proposed method to facilitate further research.