# Reversible Vision Transformers

arXiv: https://arxiv.org/abs/2302.04869 · HTML: https://mlsanity.com/paper/2302.04869/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 74 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)

## Built on by (most cited first, 30 of 33)

- [Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data](https://mlsanity.com/paper/2402.05892/) (2024)
- [Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?](https://mlsanity.com/paper/2407.21792/) (2024)
- [Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning](https://mlsanity.com/paper/2306.00477/) (2023)
- [UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memory](https://mlsanity.com/paper/2308.14316/) (2023)
- [AdaRevD: Adaptive Patch Exiting Reversible Decoder Pushes the Limit of Image Deblurring](https://mlsanity.com/paper/2406.09135/) (2024)
- [Training a Large Video Model on a Single Machine in a Day](https://mlsanity.com/paper/2309.16669/) (2023)
- [Training Neural Networks from Scratch with Parallel Low-Rank Adapters](https://mlsanity.com/paper/2402.16828/) (2024)
- [Invertible Diffusion Models for Compressed Sensing](https://mlsanity.com/paper/2403.17006/) (2024)
- [SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning](https://mlsanity.com/paper/2407.07523/) (2024)
- [Data-independent Module-aware Pruning for Hierarchical Vision Transformers](https://mlsanity.com/paper/2404.13648/) (2024)
- [PETRA: Parallel End-to-end Training with Reversible Architectures](https://mlsanity.com/paper/2406.02052/) (2024)
- [ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training](https://mlsanity.com/paper/2406.02613/) (2024)
- [Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reduction](https://mlsanity.com/paper/2403.12511/) (2024)
- [Scalable Meta-Learning via Mixed-Mode Differentiation](https://mlsanity.com/paper/2505.00793/) (2025)
- [On Surjectivity of Neural Networks: Can you elicit any behavior from your model?](https://mlsanity.com/paper/2508.19445/) (2025)
- [Dr2Net: Dynamic Reversible Dual-Residual Networks for Memory-Efficient Finetuning](https://mlsanity.com/paper/2401.04105/) (2024)
- [This Probably Looks Exactly Like That: An Invertible Prototypical Network](https://mlsanity.com/paper/2407.12200/) (2024)
- [Leaping Into Memories: Space-Time Deep Feature Synthesis](https://mlsanity.com/paper/2303.09941/) (2023)
- [Token Turing Machines are Efficient Vision Models](https://mlsanity.com/paper/2409.07613/) (2024)
- [MOOSE: Pay Attention to Temporal Dynamics for Video Understanding via Optical Flows](https://mlsanity.com/paper/2506.01119/) (2025)
- [Dynamic Granularity Matters: Rethinking Vision Transformers Beyond Fixed Patch Splitting](https://mlsanity.com/paper/2511.19021/) (2025)
- [PaReprop: Fast Parallelized Reversible Backpropagation](https://mlsanity.com/paper/2306.09342/) (2023)
- [Moonwalk: Inverse-Forward Differentiation](https://mlsanity.com/paper/2402.14212/) (2024)
- [On Exact Bit-level Reversible Transformers Without Changing Architectures](https://mlsanity.com/paper/2407.09093/) (2024)
- [Active Inference AI Systems for Scientific Discovery](https://mlsanity.com/paper/2506.21329/) (2025)
- [Towards Zero Memory Footprint Spiking Neural Network Training](https://mlsanity.com/paper/2308.08649/) (2023)
- [Dimension Mixer: Group Mixing of Input Dimensions for Efficient Function Approximation](https://mlsanity.com/paper/2311.18735/) (2023)
- [Stochastic Communication Avoidance for Recommendation Systems](https://mlsanity.com/paper/2411.01611/) (2024)
- [Memory-Efficient Training for Deep Speaker Embedding Learning in Speaker Verification](https://mlsanity.com/paper/2412.01195/) (2024)
- [A Semantic-Enhanced Heterogeneous Graph Learning Method for Flexible Objects Recognition](https://mlsanity.com/paper/2503.22079/) (2025)

## Abstract (arXiv)

We present Reversible Vision Transformers, a memory efficient architecture design for visual recognition. By decoupling the GPU memory requirement from the depth of the model, Reversible Vision Transformers enable scaling up architectures with efficient memory usage. We adapt two popular models, namely Vision Transformer and Multiscale Vision Transformers, to reversible variants and benchmark extensively across both model sizes and tasks of image classification, object detection and video classification. Reversible Vision Transformers achieve a reduced memory footprint of up to 15.5x at roughly identical model complexity, parameters and accuracy, demonstrating the promise of reversible vision transformers as an efficient backbone for hardware resource limited training regimes. Finally, we find that the additional computational burden of recomputing activations is more than overcome for deeper models, where throughput can increase up to 2.3x over their non-reversible counterparts. Full code and trained models are available at https://github.com/facebookresearch/slowfast. A simpler, easy to understand and modify version is also available at https://github.com/karttikeya/minREV