mlsanityData to 7 Oct 2026

Online Convolutional Re-parameterization

arXiv 2204.00826 ↗First posted 2 Apr 2022cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Structural re-parameterization has drawn increasing attention in various computer vision tasks. It aims at improving the performance of deep models without introducing any inference-time cost. Though efficient during inference, such models rely heavily on the complicated training-time blocks to achieve high accuracy, leading to large extra training cost. In this paper, we present online convolutional re-parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. To achieve this goal, we introduce a linear scaling layer for better optimizing the online blocks. Assisted with the reduced training cost, we also explore some more effective re-param components. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. Meanwhile, equipped with OREPA, the models outperform previous methods on ImageNet by up to +0.6%.We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks. Codes are available at https://github.com/JUGGHM/OREPA_CVPR2022 .

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 2 Apr 2022.

Builds on

36 of 51 references.

References · 36

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  9. 2017Squeeze-and-Excitation Networkscited by 4,763
  10. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  11. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  12. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
24 more
  1. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  2. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  3. 2016Pyramid Scene Parsing Networkcited by 2,472
  4. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  5. 2018Group Normalizationcited by 1,514
  6. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  7. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  8. 2021RepVGG: Making VGG-style ConvNets Great Againcited by 520
  9. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  10. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  11. 2018How Does Batch Normalization Help Optimization?cited by 355
  12. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  13. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  14. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  15. 2019ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blockscited by 178
  16. 2020FcaNet: Frequency Channel Attention Networkscited by 177
  17. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 156
  18. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  19. 2021Diverse Branch Block: Building a Convolution as an Inception-like Unitcited by 87
  20. 2018ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networkscited by 53
  21. 2020DO-Conv: Depthwise Over-parameterized Convolutional Layercited by 28
  22. 2021RepNAS: Searching for Efficient Re-parameterizing Blockscited by 11
  23. 2019Efficient Differentiable Neural Architecture Search with Meta Kernelscited by 6
  24. 2021RMNet: Equivalently Removing Residual Connection from Networkscited by 5

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 2.

5 papers.

  1. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  2. 20223D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentationcited by 94
  3. 2023Scaling Up 3D Kernels with Bayesian Frequency Re-parameterization for Medical Image Segmentationcited by 12
  4. 2022RepMode: Learning to Re-parameterize Diverse Experts for Subcellular Structure Predictioncited by 3
  5. 2023Make Explicit Calibration Implicit: Calibrate Denoiser Instead of the Noise Modelcited by 2