# Reversible Recurrent Neural Networks

arXiv: https://arxiv.org/abs/1810.10999 · HTML: https://mlsanity.com/paper/1810.10999/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2018.

## Built on by (most cited first, 23 of 23)

- [Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup](https://mlsanity.com/paper/2101.06983/) (2021)
- [End-to-End Diffusion Latent Optimization Improves Classifier Guidance](https://mlsanity.com/paper/2303.13703/) (2023)
- [UnICORNN: A recurrent model for learning very long time dependencies](https://mlsanity.com/paper/2103.05487/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Reversible Vision Transformers](https://mlsanity.com/paper/2302.04869/) (2023)
- [Fine-Tuning Pre-trained Transformers into Decaying Fast Weights](https://mlsanity.com/paper/2210.04243/) (2022)
- [Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localization](https://mlsanity.com/paper/2211.14053/) (2022)
- [Memory-Efficient Differentiable Transformer Architecture Search](https://mlsanity.com/paper/2105.14669/) (2021)
- [Invertible Diffusion Models for Compressed Sensing](https://mlsanity.com/paper/2403.17006/) (2024)
- [Generalized Lagrangian Neural Networks](https://mlsanity.com/paper/2401.03728/) (2024)
- [Duplex Sequence-to-Sequence Learning for Reversible Machine Translation](https://mlsanity.com/paper/2105.03458/) (2021)
- [Initializing LSTM internal states via manifold learning](https://mlsanity.com/paper/2104.13101/) (2021)
- [RevBiFPN: The Fully Reversible Bidirectional Feature Pyramid Network](https://mlsanity.com/paper/2206.14098/) (2022)
- [TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model](https://mlsanity.com/paper/2305.10845/) (2023)
- [On Surjectivity of Neural Networks: Can you elicit any behavior from your model?](https://mlsanity.com/paper/2508.19445/) (2025)
- [Certified Invertibility in Neural Networks via Mixed-Integer Programming](https://mlsanity.com/paper/2301.11783/) (2023)
- [Moonwalk: Inverse-Forward Differentiation](https://mlsanity.com/paper/2402.14212/) (2024)
- [CAETC: Causal Autoencoding and Treatment Conditioning for Counterfactual Estimation over Time](https://mlsanity.com/paper/2603.11565/) (2026)
- [Towards Zero Memory Footprint Spiking Neural Network Training](https://mlsanity.com/paper/2308.08649/) (2023)
- [DynaLay: An Introspective Approach to Dynamic Layer Selection for Deep Networks](https://mlsanity.com/paper/2312.12781/) (2023)
- [Reversible Deep Learning for 13C NMR in Chemoinformatics: On Structures and Spectra](https://mlsanity.com/paper/2602.03875/) (2026)
- [Finite-Horizon Fisher Memory in Two-Sided Power-Bounded Recurrent Systems](https://mlsanity.com/paper/2609.39800/) (2026)
- [Depth Memory](https://mlsanity.com/paper/2610.18809/) (2026)

## Abstract (arXiv)

Recurrent neural networks (RNNs) provide state-of-the-art performance in processing sequential data but are memory intensive to train, limiting the flexibility of RNN models which can be trained. Reversible RNNs---RNNs for which the hidden-to-hidden transition can be reversed---offer a path to reduce the memory requirements of training, as hidden states need not be stored and instead can be recomputed during backpropagation. We first show that perfectly reversible RNNs, which require no storage of the hidden activations, are fundamentally limited because they cannot forget information from their hidden state. We then provide a scheme for storing a small number of bits in order to allow perfect reversal with forgetting. Our method achieves comparable performance to traditional models while reducing the activation memory cost by a factor of 10--15. We extend our technique to attention-based sequence-to-sequence models, where it maintains performance while reducing activation memory cost by a factor of 5--10 in the encoder, and a factor of 10--15 in the decoder.