# ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks

arXiv: https://arxiv.org/abs/1811.10495 · HTML: https://mlsanity.com/paper/1811.10495/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2018.

## Built on by (most cited first, 30 of 53)

- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterization](https://mlsanity.com/paper/2303.14189/) (2023)
- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learning](https://mlsanity.com/paper/2203.06359/) (2022)
- [MobileOne: An Improved One millisecond Mobile Backbone](https://mlsanity.com/paper/2206.04040/) (2022)
- [MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training](https://mlsanity.com/paper/2311.17049/) (2023)
- [Diverse Branch Block: Building a Convolution as an Inception-like Unit](https://mlsanity.com/paper/2103.13425/) (2021)
- [What Makes Convolutional Models Great on Long Sequence Modeling?](https://mlsanity.com/paper/2210.09298/) (2022)
- [Initialization and Regularization of Factorized Neural Layers](https://mlsanity.com/paper/2105.01029/) (2021)
- [NORM: Knowledge Distillation via N-to-One Representation Matching](https://mlsanity.com/paper/2305.13803/) (2023)
- [On the Effect of Pruning on Adversarial Robustness](https://mlsanity.com/paper/2108.04890/) (2021)
- [Collapsible Linear Blocks for Super-Efficient Super Resolution](https://mlsanity.com/paper/2103.09404/) (2021)
- [RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization](https://mlsanity.com/paper/2205.05671/) (2022)
- [Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptation](https://mlsanity.com/paper/2305.00603/) (2023)
- [Adaptive Sharpness-Aware Pruning for Robust Sparse Networks](https://mlsanity.com/paper/2306.14306/) (2023)
- [Two-stream Beats One-stream: Asymmetric Siamese Network for Efficient Visual Tracking](https://mlsanity.com/paper/2503.00516/) (2025)
- [Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries](https://mlsanity.com/paper/2502.02496/) (2025)
- [RepGhost: A Hardware-Efficient Ghost Module via Re-parameterization](https://mlsanity.com/paper/2211.06088/) (2022)
- [Experts Weights Averaging: A New General Training Scheme for Vision Transformers](https://mlsanity.com/paper/2308.06093/) (2023)
- [Rapid Salient Object Detection with Difference Convolutional Neural Networks](https://mlsanity.com/paper/2507.01182/) (2025)
- [ESDMR-Net: A Lightweight Network With Expand-Squeeze and Dual Multiscale Residual Connections for Medical Image Segmentation](https://mlsanity.com/paper/2312.10585/) (2023)
- [Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search](https://mlsanity.com/paper/2104.05309/) (2021)
- [Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency](https://mlsanity.com/paper/2303.11525/) (2023)
- [Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders](https://mlsanity.com/paper/2306.05023/) (2023)
- [IPT-V2: Efficient Image Processing Transformer using Hierarchical Attentions](https://mlsanity.com/paper/2404.00633/) (2024)
- [Restructurable Activation Networks](https://mlsanity.com/paper/2208.08562/) (2022)
- [Adaptive Discovering and Merging for Incremental Novel Class Discovery](https://mlsanity.com/paper/2403.03382/) (2024)
- [Online Convolutional Re-parameterization](https://mlsanity.com/paper/2204.00826/) (2022)
- [RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers](https://mlsanity.com/paper/2505.21847/) (2025)

## Abstract (arXiv)

We introduce an approach to training a given compact network. To this end, we leverage over-parameterization, which typically improves both neural network optimization and generalization. Specifically, we propose to expand each linear layer of the compact network into multiple consecutive linear layers, without adding any nonlinearity. As such, the resulting expanded network, or ExpandNet, can be contracted back to the compact one algebraically at inference. In particular, we introduce two convolutional expansion strategies and demonstrate their benefits on several tasks, including image classification, object detection, and semantic segmentation. As evidenced by our experiments, our approach outperforms both training the compact network from scratch and performing knowledge distillation from a teacher. Furthermore, our linear over-parameterization empirically reduces gradient confusion during training and improves the network generalization.