# RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition

arXiv: https://arxiv.org/abs/2105.01883 · HTML: https://mlsanity.com/paper/2105.01883/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond](https://mlsanity.com/paper/1904.11492/) (2019)
- [Additive Margin Softmax for Face Verification](https://mlsanity.com/paper/1801.05599/) (2018)
- [The MegaFace Benchmark: 1 Million Faces for Recognition at Scale](https://mlsanity.com/paper/1512.00596/) (2015)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks](https://mlsanity.com/paper/1908.03930/) (2019)
- [Fast Algorithms for Convolutional Neural Networks](https://mlsanity.com/paper/1509.09308/) (2015)
- [Fast Training of Convolutional Networks through FFTs](https://mlsanity.com/paper/1312.5851/) (2013)
- [MEC: Memory-efficient Convolution for Deep Neural Network](https://mlsanity.com/paper/1706.06873/) (2017)

## Built on by (most cited first, 30 of 40)

- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [ResMLP: Feedforward networks for image classification with data-efficient training](https://mlsanity.com/paper/2105.03404/) (2021)
- [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias](https://mlsanity.com/paper/2106.03348/) (2021)
- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)
- [Three things everyone should know about Vision Transformers](https://mlsanity.com/paper/2203.09795/) (2022)
- [Graph-MLP: Node Classification without Message Passing in Graph](https://mlsanity.com/paper/2106.04051/) (2021)
- [PointMixer: MLP-Mixer for Point Cloud Understanding](https://mlsanity.com/paper/2111.11187/) (2021)
- [SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos](https://mlsanity.com/paper/2112.13715/) (2021)
- [A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)](https://mlsanity.com/paper/2402.07410/) (2024)
- [Augmenting Convolutional networks with attention-based aggregation](https://mlsanity.com/paper/2112.13692/) (2021)
- [ALOFT: A Lightweight MLP-like Architecture with Dynamic Low-frequency Transform for Domain Generalization](https://mlsanity.com/paper/2303.11674/) (2023)
- [On the Strong Correlation Between Model Invariance and Generalization](https://mlsanity.com/paper/2207.07065/) (2022)
- [PNVC: Towards Practical INR-based Video Compression](https://mlsanity.com/paper/2409.00953/) (2024)
- [Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report](https://mlsanity.com/paper/2211.03885/) (2022)
- [Toward a Holistic Evaluation of Robustness in CLIP Models](https://mlsanity.com/paper/2410.01534/) (2024)
- [Experts Weights Averaging: A New General Training Scheme for Vision Transformers](https://mlsanity.com/paper/2308.06093/) (2023)
- [RepNAS: Searching for Efficient Re-parameterizing Blocks](https://mlsanity.com/paper/2109.03508/) (2021)
- [4K-Resolution Photo Exposure Correction at 125 FPS with ~8K Parameters](https://mlsanity.com/paper/2311.08759/) (2023)
- [QbyE-MLPMixer: Query-by-Example Open-Vocabulary Keyword Spotting using MLPMixer](https://mlsanity.com/paper/2206.13231/) (2022)
- [ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts](https://mlsanity.com/paper/2412.08341/) (2024)
- [Arbitrary Distribution Modeling with Censorship in Real-Time Bidding Advertising](https://mlsanity.com/paper/2110.13587/) (2021)
- [MDMLP: Image Classification from Scratch on Small Datasets with MLP](https://mlsanity.com/paper/2205.14477/) (2022)
- [ASR: Attention-alike Structural Re-parameterization](https://mlsanity.com/paper/2304.06345/) (2023)
- [Learning Pixel-adaptive Multi-layer Perceptrons for Real-time Image Enhancement](https://mlsanity.com/paper/2507.12135/) (2025)
- [A channel attention based MLP-Mixer network for motor imagery decoding with EEG](https://mlsanity.com/paper/2110.10939/) (2021)
- [Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP](https://mlsanity.com/paper/2207.07284/) (2022)
- [SpiralMLP: A Lightweight Vision MLP Architecture](https://mlsanity.com/paper/2404.00648/) (2024)

## Abstract (arXiv)

We propose RepMLP, a multi-layer-perceptron-style neural network building block for image recognition, which is composed of a series of fully-connected (FC) layers. Compared to convolutional layers, FC layers are more efficient, better at modeling the long-range dependencies and positional patterns, but worse at capturing the local structures, hence usually less favored for image recognition. We propose a structural re-parameterization technique that adds local prior into an FC to make it powerful for image recognition. Specifically, we construct convolutional layers inside a RepMLP during training and merge them into the FC for inference. On CIFAR, a simple pure-MLP model shows performance very close to CNN. By inserting RepMLP in traditional CNN, we improve ResNets by 1.8% accuracy on ImageNet, 2.9% for face recognition, and 2.3% mIoU on Cityscapes with lower FLOPs. Our intriguing findings highlight that combining the global representational capacity and positional perception of FC with the local prior of convolution can improve the performance of neural network with faster speed on both the tasks with translation invariance (e.g., semantic segmentation) and those with aligned images and positional patterns (e.g., face recognition). The code and models are available at https://github.com/DingXiaoH/RepMLP.