# h-detach: Modifying the LSTM Gradient Towards Better Optimization

arXiv: https://arxiv.org/abs/1810.03023 · HTML: https://mlsanity.com/paper/1810.03023/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2018.

## Built on by (most cited first, 5 of 5)

- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Working Memory Connections for LSTM](https://mlsanity.com/paper/2109.00020/) (2021)
- [Preventing posterior collapse in variational autoencoders for text generation via decoder regularization](https://mlsanity.com/paper/2110.14945/) (2021)
- [Extending Memory for Language Modelling](https://mlsanity.com/paper/2305.11462/) (2023)
- [Detached Skip-Links and R-Probe: Decoupling Feature Aggregation from Gradient Propagation for MLLM OCR](https://mlsanity.com/paper/2603.20020/) (2026)

## Abstract (arXiv)

Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately over a large number of steps. We introduce a simple stochastic algorithm (textith-detach) that is specific to LSTM optimization and targeted towards addressing this problem. Specifically, we show that when the LSTM weights are large, the gradient components through the linear path (cell state) in the LSTM computational graph get suppressed. Based on the hypothesis that these components carry information about long term dependencies (which we show empirically), their suppression can prevent LSTMs from capturing them. Our algorithmfootnoteOur code is available at https://github.com/bhargav104/h-detach. prevents gradients flowing through this path from getting suppressed, thus allowing the LSTM to capture such dependencies better. We show significant improvements over vanilla LSTM gradient based training in terms of convergence speed, robustness to seed and learning rate, and generalization using our modification of LSTM gradient on various benchmark datasets.