# RePr: Improved Training of Convolutional Filters

arXiv: https://arxiv.org/abs/1811.07275 · HTML: https://mlsanity.com/paper/1811.07275/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2018.

## Built on by (most cited first, 12 of 12)

- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [A General Descent Aggregation Framework for Gradient-based Bi-level Optimization](https://mlsanity.com/paper/2102.07976/) (2021)
- [Knowledge Evolution in Neural Networks](https://mlsanity.com/paper/2103.05152/) (2021)
- [Re-energizing Domain Discriminator with Sample Relabeling for Adversarial Domain Adaptation](https://mlsanity.com/paper/2103.11661/) (2021)
- [An Explainable CNN Approach for Medical Codes Prediction from Clinical Text](https://mlsanity.com/paper/2101.11430/) (2021)
- [Neural Network Compression by Joint Sparsity Promotion and Redundancy Reduction](https://mlsanity.com/paper/2210.07451/) (2022)
- [Hierarchical Weight Averaging for Deep Neural Networks](https://mlsanity.com/paper/2304.11519/) (2023)
- [Weight Evolution: Improving Deep Neural Networks Training through Evolving Inferior Weight Values](https://mlsanity.com/paper/2110.04492/) (2021)
- [Compact Multi-level Sparse Neural Networks with Input Independent Dynamic Rerouting](https://mlsanity.com/paper/2112.10930/) (2021)
- [Edge Deep Learning Model Protection via Neuron Authorization](https://mlsanity.com/paper/2303.12397/) (2023)
- [Cup Curriculum: Curriculum Learning on Model Capacity](https://mlsanity.com/paper/2311.03956/) (2023)

## Abstract (arXiv)

A well-trained Convolutional Neural Network can easily be pruned without significant loss of performance. This is because of unnecessary overlap in the features captured by the network's filters. Innovations in network architecture such as skip/dense connections and Inception units have mitigated this problem to some extent, but these improvements come with increased computation and memory requirements at run-time. We attempt to address this problem from another angle - not by changing the network structure but by altering the training method. We show that by temporarily pruning and then restoring a subset of the model's filters, and repeating this process cyclically, overlap in the learned features is reduced, producing improved generalization. We show that the existing model-pruning criteria are not optimal for selecting filters to prune in this context and introduce inter-filter orthogonality as the ranking criteria to determine under-expressive filters. Our method is applicable both to vanilla convolutional networks and more complex modern architectures, and improves the performance across a variety of tasks, especially when applied to smaller networks.