# A Statistical Framework for Model Selection in LSTM Networks

arXiv: https://arxiv.org/abs/2506.06840 · HTML: https://mlsanity.com/paper/2506.06840/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 70 references that are in the map.
- First posted on arXiv 7 Jun 2025.

## Builds on (most cited first, 14 of 14)

- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Feature Engineering for Predictive Modeling using Reinforcement Learning](https://mlsanity.com/paper/1709.07150/) (2017)
- [Time Series Prediction using Deep Learning Methods in Healthcare](https://mlsanity.com/paper/2108.13461/) (2021)
- [Bayesian Recurrent Neural Networks](https://mlsanity.com/paper/1704.02798/) (2017)
- [Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective](https://mlsanity.com/paper/1908.10920/) (2019)
- [Annotated History of Modern AI and Deep Learning](https://mlsanity.com/paper/2212.11279/) (2022)
- [From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning](https://mlsanity.com/paper/1708.02478/) (2017)
- [Variational Bayes with Intractable Likelihood](https://mlsanity.com/paper/1503.08621/) (2015)
- [Structured Dropout Variational Inference for Bayesian Neural Networks](https://mlsanity.com/paper/2102.07927/) (2021)
- [A Statistical-Modelling Approach to Feedforward Neural Network Model Selection](https://mlsanity.com/paper/2207.04248/) (2022)

## Abstract (arXiv)

Long Short-Term Memory (LSTM) neural network models have become the cornerstone for sequential data modeling in numerous applications, ranging from natural language processing to time series forecasting. Despite their success, the problem of model selection, including hyperparameter tuning, architecture specification, and regularization choice remains largely heuristic and computationally expensive. In this paper, we propose a unified statistical framework for systematic model selection in LSTM networks. Our framework extends classical model selection ideas, such as information criteria and shrinkage estimation, to sequential neural networks. We define penalized likelihoods adapted to temporal structures, propose a generalized threshold approach for hidden state dynamics, and provide efficient estimation strategies using variational Bayes and approximate marginal likelihood methods. Several biomedical data centric examples demonstrate the flexibility and improved performance of the proposed framework.