# The Reversible Residual Network: Backpropagation Without Storing Activations

arXiv: https://arxiv.org/abs/1707.04585 · HTML: https://mlsanity.com/paper/1707.04585/

## Key facts

- Cited by 217 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jul 2017.

## Built on by (most cited first, 30 of 217)

- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2206.06522/) (2022)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [Training Graph Neural Networks with 1000 Layers](https://mlsanity.com/paper/2106.07476/) (2021)
- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup](https://mlsanity.com/paper/2101.06983/) (2021)
- [A Practical Survey on Faster and Lighter Transformers](https://mlsanity.com/paper/2103.14636/) (2021)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [Invertible Denoising Network: A Light Solution for Real Noise Removal](https://mlsanity.com/paper/2104.10546/) (2021)
- [A Survey on Graph Representation Learning Methods](https://mlsanity.com/paper/2204.01855/) (2022)
- [Memory-efficient Transformers via Top-k Attention](https://mlsanity.com/paper/2106.06899/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Pointerformer: Deep Reinforced Multi-Pointer Transformer for the Traveling Salesman Problem](https://mlsanity.com/paper/2304.09407/) (2023)
- [Resampling Base Distributions of Normalizing Flows](https://mlsanity.com/paper/2110.15828/) (2021)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [Momentum Residual Neural Networks](https://mlsanity.com/paper/2102.07870/) (2021)
- [The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?](https://mlsanity.com/paper/2507.08802/) (2025)
- [DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification](https://mlsanity.com/paper/2311.16124/) (2023)
- [A Survey of Techniques for Optimizing Transformer Inference](https://mlsanity.com/paper/2307.07982/) (2023)
- [Reversible Vision Transformers](https://mlsanity.com/paper/2302.04869/) (2023)
- [Reversible Column Networks](https://mlsanity.com/paper/2212.11696/) (2022)
- [Transformed Distribution Matching for Missing Value Imputation](https://mlsanity.com/paper/2302.10363/) (2023)
- [Memory-Efficient Network for Large-scale Video Compressive Sensing](https://mlsanity.com/paper/2103.03089/) (2021)
- [Self-learning Machines based on Hamiltonian Echo Backpropagation](https://mlsanity.com/paper/2103.04992/) (2021)
- [Do Residual Neural Networks discretize Neural Ordinary Differential Equations?](https://mlsanity.com/paper/2205.14612/) (2022)
- [ApiQ: Finetuning of 2-Bit Quantized Large Language Model](https://mlsanity.com/paper/2402.05147/) (2024)
- [SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit models](https://mlsanity.com/paper/2106.00553/) (2021)

## Abstract (arXiv)

Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.