# Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning

arXiv: https://arxiv.org/abs/1711.01577 · HTML: https://mlsanity.com/paper/1711.01577/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2017.

## Built on by (most cited first, 6 of 6)

- [A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold](https://mlsanity.com/paper/2302.08210/) (2023)
- [Tensor-Train Recurrent Neural Networks for Interpretable Multi-Way Financial Forecasting](https://mlsanity.com/paper/2105.04983/) (2021)
- [Optimum Output Long Short-Term Memory Cell for High-Frequency Trading Forecasting](https://mlsanity.com/paper/2304.09840/) (2023)
- [Autoencoders with Intrinsic Dimension Constraints for Learning Low Dimensional Image Representations](https://mlsanity.com/paper/2304.07686/) (2023)
- [Learning Robust and Lightweight Model through Separable Structured Transformations](https://mlsanity.com/paper/2112.13551/) (2021)
- [DeLELSTM: Decomposition-based Linear Explainable LSTM to Capture Instantaneous and Long-term Effects in Time Series](https://mlsanity.com/paper/2308.13797/) (2023)

## Abstract (arXiv)

Long Short-Term Memory (LSTM) is a popular approach to boosting the ability of Recurrent Neural Networks to store longer term temporal information. The capacity of an LSTM network can be increased by widening and adding layers. However, usually the former introduces additional parameters, while the latter increases the runtime. As an alternative we propose the Tensorized LSTM in which the hidden states are represented by tensors and updated via a cross-layer convolution. By increasing the tensor size, the network can be widened efficiently without additional parameters since the parameters are shared across different locations in the tensor; by delaying the output, the network can be deepened implicitly with little additional runtime since deep computations for each timestep are merged into temporal computations of the sequence. Experiments conducted on five challenging sequence learning tasks show the potential of the proposed model.