# Reversible Column Networks

arXiv: https://arxiv.org/abs/2212.11696 · HTML: https://mlsanity.com/paper/2212.11696/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 84 references that are in the map.
- First posted on arXiv 22 Dec 2022.

## Builds on (most cited first, 30 of 64)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)

## Built on by (most cited first, 30 of 32)

- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection](https://mlsanity.com/paper/2303.11926/) (2023)
- [ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities](https://mlsanity.com/paper/2305.11172/) (2023)
- [ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions](https://mlsanity.com/paper/2403.07392/) (2024)
- [GridFormer: Residual Dense Transformer with Grid Structure for Image Restoration in Adverse Weather Conditions](https://mlsanity.com/paper/2305.17863/) (2023)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [AdaRevD: Adaptive Patch Exiting Reversible Decoder Pushes the Limit of Image Deblurring](https://mlsanity.com/paper/2406.09135/) (2024)
- [Reversible Decoupling Network for Single Image Reflection Removal](https://mlsanity.com/paper/2410.08063/) (2024)
- [Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection](https://mlsanity.com/paper/2310.15670/) (2023)
- [DenseNets Reloaded: Paradigm Shift Beyond ResNets and ViTs](https://mlsanity.com/paper/2403.19588/) (2024)
- [Auto-Regressively Generating Multi-View Consistent Images](https://mlsanity.com/paper/2506.18527/) (2025)
- [S4DL: Shift-sensitive Spatial-Spectral Disentangling Learning for Hyperspectral Image Unsupervised Domain Adaptation](https://mlsanity.com/paper/2408.15263/) (2024)
- [On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes](https://mlsanity.com/paper/2410.19553/) (2024)
- [The Power of Linear Combinations: Learning with Random Convolutions](https://mlsanity.com/paper/2301.11360/) (2023)
- [RevColV2: Exploring Disentangled Representations in Masked Image Modeling](https://mlsanity.com/paper/2309.01005/) (2023)
- [SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement](https://mlsanity.com/paper/2507.01643/) (2025)
- [Universal Image Restoration Pre-training via Masked Degradation Classification](https://mlsanity.com/paper/2510.13282/) (2025)
- [CEDNet: A Cascade Encoder-Decoder Network for Dense Prediction](https://mlsanity.com/paper/2302.06052/) (2023)
- [Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition](https://mlsanity.com/paper/2407.14302/) (2024)
- [The Missing Point in Vision Transformers for Universal Image Segmentation](https://mlsanity.com/paper/2505.19795/) (2025)
- [LeYOLO, New Embedded Architecture for Object Detection](https://mlsanity.com/paper/2406.14239/) (2024)
- [FedPartWhole: Federated domain generalization via consistent part-whole hierarchies](https://mlsanity.com/paper/2407.14792/) (2024)
- [Explicitly Modeling Pre-Cortical Vision with a Neuro-Inspired Front-End Improves CNN Robustness](https://mlsanity.com/paper/2409.16838/) (2024)
- [A Recipe for Efficient SBIR Models: Combining Relative Triplet Loss with Batch Normalization and Knowledge Distillation](https://mlsanity.com/paper/2305.18988/) (2023)
- [Chanel-Orderer: A Channel-Ordering Predictor for Tri-Channel Natural Images](https://mlsanity.com/paper/2411.13021/) (2024)
- [Identifying Surgical Instruments in Pedagogical Cataract Surgery Videos through an Optimized Aggregation Network](https://mlsanity.com/paper/2501.02618/) (2025)
- [CascadedViT: Cascaded Chunk-FeedForward and Cascaded Group Attention Vision Transformer](https://mlsanity.com/paper/2511.14111/) (2025)

## Abstract (arXiv)

We propose a new neural network design paradigm Reversible Column Network (RevCol). The main body of RevCol is composed of multiple copies of subnetworks, named columns respectively, between which multi-level reversible connections are employed. Such architectural scheme attributes RevCol very different behavior from conventional networks: during forward propagation, features in RevCol are learned to be gradually disentangled when passing through each column, whose total information is maintained rather than compressed or discarded as other network does. Our experiments suggest that CNN-style RevCol models can achieve very competitive performances on multiple computer vision tasks such as image classification, object detection and semantic segmentation, especially with large parameter budget and large dataset. For example, after ImageNet-22K pre-training, RevCol-XL obtains 88.2% ImageNet-1K accuracy. Given more pre-training data, our largest model RevCol-H reaches 90.0% on ImageNet-1K, 63.8% APbox on COCO detection minival set, 61.0% mIoU on ADE20k segmentation. To our knowledge, it is the best COCO detection and ADE20k segmentation result among pure (static) CNN models. Moreover, as a general macro architecture fashion, RevCol can also be introduced into transformers or other neural networks, which is demonstrated to improve the performances in both computer vision and NLP tasks. We release code and models at https://github.com/megvii-research/RevCol