mlsanityData to 7 Oct 2026

The Reversible Residual Network: Backpropagation Without Storing Activations

arXiv 1707.04585 ↗First posted 14 Jul 2017cs.CV cs.LGcited by 215

Not on the map. Why?

Abstract (arXiv)

Deep residual networks (ResNets) have significantly pushed forward the state-of-the-art on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck, as one needs to store the activations in order to calculate gradients using backpropagation. We present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backpropagation. We demonstrate the effectiveness of RevNets on CIFAR-10, CIFAR-100, and ImageNet, establishing nearly identical classification accuracy to equally-sized ResNets, even though the activation storage requirements are independent of depth.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 31; 2023: 42; 2024: 36; 2025: 33; 2026: 22.

215 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 422
  3. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  4. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  5. 2022CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusioncited by 222
  6. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  7. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  8. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  9. 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 123
  10. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  11. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  12. 2021Invertible Denoising Network: A Light Solution for Real Noise Removalcited by 68
38 more
  1. 2022A Survey on Graph Representation Learning Methodscited by 66
  2. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  3. 2021A Survey on Green Deep Learningcited by 52
  4. 2023Pointerformer: Deep Reinforced Multi-Pointer Transformer for the Traveling Salesman Problemcited by 49
  5. 2021Resampling Base Distributions of Normalizing Flowscited by 42
  6. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  7. 2021Momentum Residual Neural Networkscited by 36
  8. 2023DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purificationcited by 36
  9. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  10. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  11. 2023Reversible Vision Transformerscited by 33
  12. 2022Reversible Column Networkscited by 32
  13. 2023Transformed Distribution Matching for Missing Value Imputationcited by 31
  14. 2021Memory-Efficient Network for Large-scale Video Compressive Sensingcited by 30
  15. 2021Self-learning Machines based on Hamiltonian Echo Backpropagationcited by 28
  16. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  17. 2024ApiQ: Finetuning of 2-Bit Quantized Large Language Modelcited by 27
  18. 2021SHINE: SHaring the INverse Estimate from the forward pass for bi-level optimization and implicit modelscited by 26
  19. 2023Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuningcited by 26
  20. 2023UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memorycited by 26
  21. 2024AdaRevD: Adaptive Patch Exiting Reversible Decoder Pushes the Limit of Image Deblurringcited by 25
  22. 2021No frame left behind: Full Video Action Recognitioncited by 24
  23. 2023CAP-VSTNet: Content Affinity Preserved Versatile Style Transfercited by 24
  24. 2024Neural Combinatorial Optimization Algorithms for Solving Vehicle Routing Problems: A Comprehensive Survey with Perspectivescited by 23
  25. 2024Reversible Decoupling Network for Single Image Reflection Removalcited by 22
  26. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  27. 2023Training a Large Video Model on a Single Machine in a Daycited by 20
  28. 2022MGNNI: Multiscale Graph Neural Networks with Implicit Layerscited by 19
  29. 2022Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localizationcited by 19
  30. 2021Do We Need Anisotropic Graph Neural Networks?cited by 18
  31. 2021Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphscited by 18
  32. 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
  33. 2021An Information Theory-inspired Strategy for Automatic Network Pruningcited by 17
  34. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  35. 2021The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architecturescited by 16
  36. 2023DisCo-CLIP: A Distributed Contrastive Loss for Memory Efficient CLIP Trainingcited by 15
  37. 2021Memory-Efficient Differentiable Transformer Architecture Searchcited by 13
  38. 2021Universal Joint Approximation of Manifolds and Densities by Simple Injective Flowscited by 13

50 most cited of 215 All 215 (JSON)