# The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures

arXiv: https://arxiv.org/abs/2006.16242 · HTML: https://mlsanity.com/paper/2006.16242/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2020.

## Built on by (most cited first, 13 of 13)

- [The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2404.10343/) (2024)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [Blueprint Separable Residual Network for Efficient Image Super-Resolution](https://mlsanity.com/paper/2205.05996/) (2022)
- [NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results](https://mlsanity.com/paper/2205.05675/) (2022)
- [NTIRE 2024 Challenge on Image Super-Resolution (x4): Methods and Results](https://mlsanity.com/paper/2404.09790/) (2024)
- [CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolution](https://mlsanity.com/paper/2207.10345/) (2022)
- [Towards Efficient Graph Convolutional Networks for Point Cloud Handling](https://mlsanity.com/paper/2104.05706/) (2021)
- [Edge-enhanced Feature Distillation Network for Efficient Super-Resolution](https://mlsanity.com/paper/2204.08759/) (2022)
- [AdaBM: On-the-Fly Adaptive Bit Mapping for Image Super-Resolution](https://mlsanity.com/paper/2404.03296/) (2024)
- [Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks](https://mlsanity.com/paper/2307.13337/) (2023)
- [IMDeception: Grouped Information Distilling Super-Resolution Network](https://mlsanity.com/paper/2204.11463/) (2022)
- [Learning to Forget using Hypernetworks](https://mlsanity.com/paper/2412.00761/) (2024)
- [One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression](https://mlsanity.com/paper/2508.13836/) (2025)

## Abstract (arXiv)

In this paper, we tackle the problem of convolutional neural network design. Instead of focusing on the design of the overall architecture, we investigate a design space that is usually overlooked, i.e. adjusting the channel configurations of predefined networks. We find that this adjustment can be achieved by shrinking widened baseline networks and leads to superior performance. Based on that, we articulate the heterogeneity hypothesis: with the same training protocol, there exists a layer-wise differentiated network architecture (LW-DNA) that can outperform the original network with regular channel configurations but with a lower level of model complexity. The LW-DNA models are identified without extra computational cost or training time compared with the original network. This constraint leads to controlled experiments which direct the focus to the importance of layer-wise specific channel configurations. LW-DNA models come with advantages related to overfitting, i.e. the relative relationship between model complexity and dataset size. Experiments are conducted on various networks and datasets for image classification, visual tracking and image restoration. The resultant LW-DNA models consistently outperform the baseline models. Code is available at https://github.com/ofsoundof/Heterogeneity_Hypothesis.