Gated Compression Layers for Efficient Always-On Models
Not on the map. Why?
Abstract (arXiv)
Mobile and embedded machine learning developers frequently have to compromise between two inferior on-device deployment strategies: sacrifice accuracy and aggressively shrink their models to run on dedicated low-power cores; or sacrifice battery by running larger models on more powerful compute cores such as neural processing units or the main application processor. In this paper, we propose a novel Gated Compression layer that can be applied to transform existing neural network architectures into Gated Neural Networks. Gated Neural Networks have multiple properties that excel for on-device use cases that help significantly reduce power, boost accuracy, and take advantage of heterogeneous compute cores. We provide results across five public image and audio datasets that demonstrate the proposed Gated Compression layer effectively stops up to 96% of negative samples, compresses 97% of positive samples, while maintaining or improving model accuracy.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 43 of its 50 references that are in the map. First posted on arXiv 15 Mar 2023.
Builds on
43 of 50 references.
References · 43
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2014Going Deeper with Convolutionscited by 4,231
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
31 more
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2019Searching for MobileNetV3cited by 1,984
- 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
- 2017Mixed Precision Trainingcited by 1,135
- 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
- 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
- 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
- 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
- 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
- 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 426
- 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
- 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 225
- 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
- 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
- 2020TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systemscited by 161
- 2015Compressing Neural Networks with the Hashing Trickcited by 150
- 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 121
- 2017Distributed Deep Neural Networks over the Cloud, the Edge and End Devicescited by 119
- 2016Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1cited by 102
- 2018Federated Learning for Keyword Spottingcited by 95
- 2015Fast ConvNets Using Group-wise Brain Damagecited by 81
- 2018A Survey on Methods and Theories of Quantized Neural Networkscited by 75
- 2019On-Device Machine Learning: An Algorithms and Learning Theory Perspectivecited by 63
- 2018SqueezeNext: Hardware-Aware Neural Network Designcited by 61
- 2020Pruning Algorithms to Accelerate Convolutional Neural Networks for Edge Applications: A Surveycited by 26
- 2020Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networkscited by 23
- 2020Learnable Bernoulli Dropout for Bayesian Deep Learningcited by 14
- 2017ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projectionscited by 11
- 2022Class Based Thresholding in Early Exit Semantic Segmentation Networkscited by 5
- 2022Pruning Early Exit Networkscited by 4
- 2015Reduced-Precision Strategies for Bounded Memory in Deep Neural Netscited by 3
Built on by
Citing papers per year: 2024: 2.
2 papers.