# Factorization tricks for LSTM networks

arXiv: https://arxiv.org/abs/1703.10722 · HTML: https://mlsanity.com/paper/1703.10722/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2017.

## Built on by (most cited first, 13 of 13)

- [Federated Learning Meets Natural Language Processing: A Survey](https://mlsanity.com/paper/2107.12603/) (2021)
- [Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN](https://mlsanity.com/paper/2106.12901/) (2021)
- [Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling](https://mlsanity.com/paper/2402.10211/) (2024)
- [A new hope for network model generalization](https://mlsanity.com/paper/2207.05843/) (2022)
- [Greenformer: Factorization Toolkit for Efficient Deep Neural Networks](https://mlsanity.com/paper/2109.06762/) (2021)
- [Generative AI Agents in Autonomous Machines: A Safety Perspective](https://mlsanity.com/paper/2410.15489/) (2024)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation](https://mlsanity.com/paper/2108.10808/) (2021)
- [Attention-based Contextual Language Model Adaptation for Speech Recognition](https://mlsanity.com/paper/2106.01451/) (2021)
- [A Survey of Distributed Learning in Cloud, Mobile, and Edge Settings](https://mlsanity.com/paper/2405.15079/) (2024)
- [IMSSA: Deploying modern state-space models on memristive in-memory compute hardware](https://mlsanity.com/paper/2412.20215/) (2024)
- [Adversarially Robust and Explainable Model Compression with On-Device Personalization for Text Classification](https://mlsanity.com/paper/2101.05624/) (2021)
- [Extending Memory for Language Modelling](https://mlsanity.com/paper/2305.11462/) (2023)

## Abstract (arXiv)

We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups. Both approaches allow us to train large LSTM networks significantly faster to the near state-of the art perplexity while using significantly less RNN parameters.