# Towards Binary-Valued Gates for Robust LSTM Training

arXiv: https://arxiv.org/abs/1806.02988 · HTML: https://mlsanity.com/paper/1806.02988/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jun 2018.

## Built on by (most cited first, 3 of 3)

- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [Sparsely Changing Latent States for Prediction and Planning in Partially Observable Domains](https://mlsanity.com/paper/2110.15949/) (2021)
- [National-scale electricity peak load forecasting: Traditional, machine learning, or hybrid model?](https://mlsanity.com/paper/2107.06174/) (2021)

## Abstract (arXiv)

Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling. It aims to use gates to control information flow (e.g., whether to skip some information or not) in the recurrent computations, although its practical implementation based on soft gates only partially achieves this goal. In this paper, we propose a new way for LSTM training, which pushes the output values of the gates towards 0 or 1. By doing so, we can better control the information flow: the gates are mostly open or closed, instead of in a middle state, which makes the results more interpretable. Empirical studies show that (1) Although it seems that we restrict the model capacity, there is no performance drop: we achieve better or comparable performances due to its better generalization ability; (2) The outputs of gates are not sensitive to their inputs: we can easily compress the LSTM unit in multiple ways, e.g., low-rank approximation and low-precision approximation. The compressed models are even better than the baseline models without compression.