# Channel Equilibrium Networks for Learning Deep Representation

arXiv: https://arxiv.org/abs/2003.00214 · HTML: https://mlsanity.com/paper/2003.00214/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Feb 2020.

## Built on by (most cited first, 3 of 3)

- [QuadTree Attention for Vision Transformers](https://mlsanity.com/paper/2201.02767/) (2022)
- [Channel Exchanging Networks for Multimodal and Multitask Dense Image Prediction](https://mlsanity.com/paper/2112.02252/) (2021)
- [Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation](https://mlsanity.com/paper/2204.09847/) (2022)

## Abstract (arXiv)

Convolutional Neural Networks (CNNs) are typically constructed by stacking multiple building blocks, each of which contains a normalization layer such as batch normalization (BN) and a rectified linear function such as ReLU. However, this work shows that the combination of normalization and rectified linear function leads to inhibited channels, which have small magnitude and contribute little to the learned feature representation, impeding the generalization ability of CNNs. Unlike prior arts that simply removed the inhibited channels, we propose to "wake them up" during training by designing a novel neural building block, termed Channel Equilibrium (CE) block, which enables channels at the same layer to contribute equally to the learned representation. We show that CE is able to prevent inhibited channels both empirically and theoretically. CE has several appealing benefits. (1) It can be integrated into many advanced CNN architectures such as ResNet and MobileNet, outperforming their original networks. (2) CE has an interesting connection with the Nash Equilibrium, a well-known solution of a non-cooperative game. (3) Extensive experiments show that CE achieves state-of-the-art performance on various challenging benchmarks such as ImageNet and COCO.