# Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks

arXiv: https://arxiv.org/abs/1707.06799 · HTML: https://mlsanity.com/paper/1707.06799/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2017.

## Built on by (most cited first, 17 of 17)

- [Data Augmentation techniques in time series domain: A survey and taxonomy](https://mlsanity.com/paper/2206.13508/) (2022)
- [CitationIE: Leveraging the Citation Graph for Scientific Information Extraction](https://mlsanity.com/paper/2106.01560/) (2021)
- [Benchmarking Online Sequence-to-Sequence and Character-based Handwriting Recognition from IMU-Enhanced Pens](https://mlsanity.com/paper/2202.07036/) (2022)
- [Linguistically-Enriched and Context-Aware Zero-shot Slot Filling](https://mlsanity.com/paper/2101.06514/) (2021)
- [Multi-annotator Deep Learning: A Probabilistic Framework for Classification](https://mlsanity.com/paper/2304.02539/) (2023)
- [LaTeX-Numeric: Language-agnostic Text attribute eXtraction for E-commerce Numeric Attributes](https://mlsanity.com/paper/2104.09576/) (2021)
- [Natural Language Instructions for Intuitive Human Interaction with Robotic Assistants in Field Construction Work](https://mlsanity.com/paper/2307.04195/) (2023)
- [De-identifying Australian Hospital Discharge Summaries: An End-to-End Framework using Ensemble of Deep Learning Models](https://mlsanity.com/paper/2101.00146/) (2021)
- [Toward Open-domain Slot Filling via Self-supervised Co-training](https://mlsanity.com/paper/2303.13801/) (2023)
- [Benchmarking Modern Named Entity Recognition Techniques for Free-text Health Record De-identification](https://mlsanity.com/paper/2103.13546/) (2021)
- [LSTM Hyper-Parameter Selection for Malware Detection: Interaction Effects and Hierarchical Selection Approach](https://mlsanity.com/paper/2109.11500/) (2021)
- [On Mini-Batch Training with Varying Length Time Series](https://mlsanity.com/paper/2212.06536/) (2022)
- [Introducing "Forecast Utterance" for Conversational Data Science](https://mlsanity.com/paper/2309.03877/) (2023)
- [Temporal Multi-Signal Fusion for Token-Level Hallucination Detection](https://mlsanity.com/paper/2608.18115/) (2026)
- [Short-term daily precipitation forecasting with seasonally-integrated autoencoder](https://mlsanity.com/paper/2101.09509/) (2021)
- [ICU Mortality Prediction Using Long Short-Term Memory Networks](https://mlsanity.com/paper/2308.12800/) (2023)
- [Adversarial Attacks on Online Handwriting using Salience-based Temporal Editing](https://mlsanity.com/paper/2607.12500/) (2026)

## Abstract (arXiv)

Selecting optimal parameters for a neural network architecture can often make the difference between mediocre and state-of-the-art performance. However, little is published which parameters and design choices should be evaluated or selected making the correct hyperparameter optimization often a "black art that requires expert experiences" (Snoek et al., 2012). In this paper, we evaluate the importance of different network design choices and hyperparameters for five common linguistic sequence tagging tasks (POS, Chunking, NER, Entity Recognition, and Event Detection). We evaluated over 50.000 different setups and found, that some parameters, like the pre-trained word embeddings or the last layer of the network, have a large impact on the performance, while other parameters, for example the number of LSTM layers or the number of recurrent units, are of minor importance. We give a recommendation on a configuration that performs well among different tasks.