mlsanityData to 7 Oct 2026

Rescaling CNN through Learnable Repetition of Network Parameters

arXiv 2101.05650 ↗First posted 14 Jan 2021cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Deeper and wider CNNs are known to provide improved performance for deep learning tasks. However, most such networks have poor performance gain per parameter increase. In this paper, we investigate whether the gain observed in deeper models is purely due to the addition of more optimization parameters or whether the physical size of the network as well plays a role. Further, we present a novel rescaling strategy for CNNs based on learnable repetition of its parameters. Based on this strategy, we rescale CNNs without changing their parameter count, and show that learnable sharing of weights itself can provide significant boost in the performance of any given model without changing its parameter count. We show that small base networks when rescaled, can provide performance comparable to deeper networks with as low as 6% of optimization parameters of the deeper one. The relevance of weight sharing is further highlighted through the example of group-equivariant CNNs. We show that the significant improvements obtained with group-equivariant CNNs over the regular CNNs on classification problems are only partly due to the added equivariance property, and part of it comes from the learnable repetition of network weights. For rot-MNIST dataset, we show that up to 40% of the relative gain reported by state-of-the-art methods for rotation equivariance could actually be due to just the learnt repetition of weights.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 14 Jan 2021.

Builds on

21 of 28 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2016Wide Residual Networkscited by 3,057
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  8. 2016Group Equivariant Convolutional Networkscited by 1,154
  9. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  10. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  11. 2019General E(2)-Equivariant Steerable CNNscited by 441
  12. 2016Steerable CNNscited by 298
9 more
  1. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  2. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  3. 2017Learning Steerable Filters for Rotation Equivariant CNNscited by 193
  4. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  5. 2018Compressing Neural Networks using the Variational Information Bottleneckcited by 59
  6. 2018Structured Pruning of Neural Networks with Budget-Aware Regularizationcited by 26
  7. 2017ShaResNet: reducing residual network parameter number by sharing weightscited by 5
  8. 2020Training highly effective connectivities within neural networks with randomly initialized, fixed weightscited by 2
  9. 2019Deep Anchored Convolutional Neural Networkscited by 1

Built on by

Not cited yet.