# Batch-Shaping for Learning Conditional Channel Gated Networks

arXiv: https://arxiv.org/abs/1907.06627 · HTML: https://mlsanity.com/paper/1907.06627/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2019.

## Built on by (most cited first, 30 of 42)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Not All Layers of LLMs Are Necessary During Inference](https://mlsanity.com/paper/2403.02181/) (2024)
- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [Learn to Match: Automatic Matching Network Design for Visual Tracking](https://mlsanity.com/paper/2108.00803/) (2021)
- [FrameExit: Conditional Early Exiting for Efficient Video Recognition](https://mlsanity.com/paper/2104.13400/) (2021)
- [Learning to Weight Samples for Dynamic Early-exiting Networks](https://mlsanity.com/paper/2209.08310/) (2022)
- [Dynamic Perceiver for Efficient Visual Recognition](https://mlsanity.com/paper/2306.11248/) (2023)
- [Skip-Convolutions for Efficient Video Processing](https://mlsanity.com/paper/2104.11487/) (2021)
- [Latency-aware Unified Dynamic Networks for Efficient Image Recognition](https://mlsanity.com/paper/2308.15949/) (2023)
- [Latency-aware Spatial-wise Dynamic Networks](https://mlsanity.com/paper/2210.06223/) (2022)
- [Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism](https://mlsanity.com/paper/2205.03055/) (2022)
- [MSViT: Dynamic Mixed-Scale Tokenization for Vision Transformers](https://mlsanity.com/paper/2307.02321/) (2023)
- [Generative Zero-shot Network Quantization](https://mlsanity.com/paper/2101.08430/) (2021)
- [Referring Expression Comprehension Using Language Adaptive Inference](https://mlsanity.com/paper/2306.04451/) (2023)
- [Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling](https://mlsanity.com/paper/2110.03252/) (2021)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Human-Object Interaction Detection via Weak Supervision](https://mlsanity.com/paper/2112.00492/) (2021)
- [Fast Conditional Network Compression Using Bayesian HyperNetworks](https://mlsanity.com/paper/2205.06404/) (2022)
- [Energy-Latency Attacks: A New Adversarial Threat to Deep Learning](https://mlsanity.com/paper/2503.04963/) (2025)
- [Auto-Compressing Subset Pruning for Semantic Image Segmentation](https://mlsanity.com/paper/2201.11103/) (2022)
- [Progressive Channel-Shrinking Network](https://mlsanity.com/paper/2304.00280/) (2023)
- [Data-Free Dynamic Compression of CNNs for Tractable Efficiency](https://mlsanity.com/paper/2309.17211/) (2023)
- [Efficient Transformer-based 3D Object Detection with Dynamic Token Halting](https://mlsanity.com/paper/2303.05078/) (2023)
- [SlowFormer: Universal Adversarial Patch for Attack on Compute and Energy Efficiency of Inference Efficient Vision Transformers](https://mlsanity.com/paper/2310.02544/) (2023)
- [CNN Mixture-of-Depths](https://mlsanity.com/paper/2409.17016/) (2024)
- [NeuralFuse: Learning to Recover the Accuracy of Access-Limited Neural Network Inference in Low-Voltage Regimes](https://mlsanity.com/paper/2306.16869/) (2023)
- [Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs](https://mlsanity.com/paper/2510.01185/) (2025)
- [Prioritized Subnet Sampling for Resource-Adaptive Supernet Training](https://mlsanity.com/paper/2109.05432/) (2021)
- [Learning a Consensus Sub-Network with Polarization Regularization and One Pass Training](https://mlsanity.com/paper/2302.10798/) (2023)

## Abstract (arXiv)

We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost. We achieve this by gating the deep-learning architecture on a fine-grained-level. Individual convolutional maps are turned on/off conditionally on features in the network. To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner. We also introduce a generally applicable tool batch-shaping that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution. We use this novel technique to force gates to be more conditional on the data. We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation. Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy. In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity. We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.