# VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks

arXiv: https://arxiv.org/abs/1909.04485 · HTML: https://mlsanity.com/paper/1909.04485/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Sep 2019.

## Built on by (most cited first, 9 of 9)

- [Structural Pruning via Latency-Saliency Knapsack](https://mlsanity.com/paper/2210.06659/) (2022)
- [When to Prune? A Policy towards Early Structural Pruning](https://mlsanity.com/paper/2110.12007/) (2021)
- [Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey](https://mlsanity.com/paper/2405.01725/) (2024)
- [HALP: Hardware-Aware Latency Pruning](https://mlsanity.com/paper/2110.10811/) (2021)
- [Feature Flow Regularization: Improving Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/2106.02914/) (2021)
- [Neural Network Panning: Screening the Optimal Sparse Network Before Training](https://mlsanity.com/paper/2209.13378/) (2022)
- [A Masked Pruning Approach for Dimensionality Reduction in Communication-Efficient Federated Learning Systems](https://mlsanity.com/paper/2312.03889/) (2023)
- [Subspace Node Pruning](https://mlsanity.com/paper/2405.17506/) (2024)
- [Using deep learning for predicting cleansing quality of colon capsule endoscopy images](https://mlsanity.com/paper/2601.13412/) (2026)

## Abstract (arXiv)

Improving weight sparsity is a common strategy for producing light-weight deep neural networks. However, pruning models with residual learning is more challenging. In this paper, we introduce Variance-Aware Cross-Layer (VACL), a novel approach to address this problem. VACL consists of two parts, a Cross-Layer grouping and a Variance Aware regularization. In Cross-Layer grouping the ith filters of layers connected by skip-connections are grouped into one regularization group. Then, the Variance-Aware regularization term takes into account both the first and second-order statistics of the connected layers to constrain the variance within a group. Our approach can effectively improve the structural sparsity of residual models. For CIFAR10, the proposed method reduces a ResNet model by up to 79.5% with no accuracy drop and reduces a ResNeXt model by up to 82% with less than 1% accuracy drop. For ImageNet, it yields a pruned ratio of up to 63.3% with less than 1% top-5 accuracy drop. Our experimental results show that the proposed approach significantly outperforms other state-of-the-art methods in terms of overall model size and accuracy.