# Convolutional Networks with Adaptive Inference Graphs

arXiv: https://arxiv.org/abs/1711.11503 · HTML: https://mlsanity.com/paper/1711.11503/

## Key facts

- Cited by 148 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Nov 2017.

## Built on by (most cited first, 30 of 148)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2105.15075/) (2021)
- [MOOD: Multi-level Out-of-distribution Detection](https://mlsanity.com/paper/2104.14726/) (2021)
- [Dynamic Slimmable Network](https://mlsanity.com/paper/2103.13258/) (2021)
- [A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning](https://mlsanity.com/paper/2110.01515/) (2021)
- [Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions](https://mlsanity.com/paper/2106.05022/) (2021)
- [Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization](https://mlsanity.com/paper/2305.11718/) (2023)
- [FrameExit: Conditional Early Exiting for Efficient Video Recognition](https://mlsanity.com/paper/2104.13400/) (2021)
- [Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification](https://mlsanity.com/paper/2412.00876/) (2024)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Adaptive Rotated Convolution for Rotated Object Detection](https://mlsanity.com/paper/2303.07820/) (2023)
- [AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition](https://mlsanity.com/paper/2102.05775/) (2021)
- [Glance and Focus Networks for Dynamic Visual Recognition](https://mlsanity.com/paper/2201.03014/) (2022)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [Towards Compact CNNs via Collaborative Compression](https://mlsanity.com/paper/2105.11228/) (2021)
- [Dynamic Network Quantization for Efficient Video Inference](https://mlsanity.com/paper/2108.10394/) (2021)
- [FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy](https://mlsanity.com/paper/2307.01217/) (2023)
- [SpaceByte: Towards Deleting Tokenization from Large Language Modeling](https://mlsanity.com/paper/2404.14408/) (2024)
- [Dynamic Perceiver for Efficient Visual Recognition](https://mlsanity.com/paper/2306.11248/) (2023)
- [Skip-Convolutions for Efficient Video Processing](https://mlsanity.com/paper/2104.11487/) (2021)
- [Dynamic Resolution Network](https://mlsanity.com/paper/2106.02898/) (2021)
- [Deep Joint Source-Channel Coding for Wireless Image Transmission with Adaptive Rate Control](https://mlsanity.com/paper/2110.04456/) (2021)
- [Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data](https://mlsanity.com/paper/2112.00054/) (2021)
- [Latency-aware Unified Dynamic Networks for Efficient Image Recognition](https://mlsanity.com/paper/2308.15949/) (2023)
- [Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks](https://mlsanity.com/paper/2109.12872/) (2021)
- [Latency-aware Spatial-wise Dynamic Networks](https://mlsanity.com/paper/2210.06223/) (2022)
- [Mixture of Nested Experts: Adaptive Processing of Visual Tokens](https://mlsanity.com/paper/2407.19985/) (2024)

## Abstract (arXiv)

Do convolutional networks really need a fixed feed-forward structure? What if, after identifying the high-level concept of an image, a network could move directly to a layer that can distinguish fine-grained differences? Currently, a network would first need to execute sometimes hundreds of intermediate layers that specialize in unrelated aspects. Ideally, the more a network already knows about an image, the better it should be at deciding which layer to compute next. In this work, we propose convolutional networks with adaptive inference graphs (ConvNet-AIG) that adaptively define their network topology conditioned on the input image. Following a high-level structure similar to residual networks (ResNets), ConvNet-AIG decides for each input image on the fly which layers are needed. In experiments on ImageNet we show that ConvNet-AIG learns distinct inference graphs for different categories. Both ConvNet-AIG with 50 and 101 layers outperform their ResNet counterpart, while using 20% and 38% less computations respectively. By grouping parameters into layers for related classes and only executing relevant layers, ConvNet-AIG improves both efficiency and overall classification quality. Lastly, we also study the effect of adaptive inference graphs on the susceptibility towards adversarial examples. We observe that ConvNet-AIG shows a higher robustness than ResNets, complementing other known defense mechanisms.