Conditional Information Gain Trellis
Not on the map. Why?
Abstract (arXiv)
Conditional computing processes an input using only part of the neural network's computational units. Learning to execute parts of a deep convolutional network by routing individual samples has several advantages: Reducing the computational burden is an obvious advantage. Furthermore, if similar classes are routed to the same path, that part of the network learns to discriminate between finer differences and better classification accuracies can be attained with fewer parameters. Recently, several papers have exploited this idea to take a particular child of a node in a tree-shaped network or to skip parts of a network. In this work, we follow a Trellis-based approach for generating specific execution paths in a deep convolutional neural network. We have designed routing mechanisms that use differentiable information gain-based cost functions to determine which subset of features in a convolutional layer will be executed. We call our method Conditional Information Gain Trellis (CIGT). We show that our conditional execution mechanism achieves comparable or better model performance compared to unconditional baselines, using only a fraction of the computational resources.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 13 Feb 2024.
Builds on
32 of 52 references.
References · 32
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2018CBAM: Convolutional Block Attention Modulecited by 2,463
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
- 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
- 2021Dynamic Neural Networks: A Surveycited by 405
20 more
- 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 364
- 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
- 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
- 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
- 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
- 2015Conditional Computation in Neural Networks for faster modelscited by 166
- 2017Convolutional Networks with Adaptive Inference Graphscited by 147
- 2013Deep Learning of Representations: Looking Forwardcited by 84
- 2015Fast ConvNets Using Group-wise Brain Damagecited by 81
- 2018Adaptive Neural Treescited by 64
- 2018Modular Networks: Learning to Decompose Neural Computationcited by 62
- 2017Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Executioncited by 59
- 2022Learning to Weight Samples for Dynamic Early-exiting Networkscited by 44
- 2017Deciding How to Decide: Dynamic Routing in Artificial Neural Networkscited by 33
- 2016Neural Random Forestscited by 25
- 2016Decision Forests, Convolutional Networks and the Models in-Betweencited by 15
- 2015Blockout: Dynamic Model Selection for Hierarchical Deep Networkscited by 11
- 2022Automatic Sparse Connectivity Learning for Neural Networkscited by 4
- 2017Deep Convolutional Decision Jungle for Image Classificationcited by 2
- 2018Conditional Information Gain Networkscited by 2
Built on by
Not cited yet.