# Efficient IoT Inference via Context-Awareness

arXiv: https://arxiv.org/abs/2310.19112 · HTML: https://mlsanity.com/paper/2310.19112/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 57 references that are in the map.
- First posted on arXiv 29 Oct 2023.

## Builds on (most cited first, 29 of 29)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [Better Aggregation in Test-Time Augmentation](https://mlsanity.com/paper/2011.11156/) (2020)
- [CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs](https://mlsanity.com/paper/1801.06601/) (2018)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [Direct Uncertainty Prediction for Medical Second Opinions](https://mlsanity.com/paper/1807.01771/) (2018)
- [Deep feature compression for collaborative object detection](https://mlsanity.com/paper/1802.03931/) (2018)
- [Focus: Querying Large Video Datasets with Low Latency and Low Cost](https://mlsanity.com/paper/1801.03493/) (2018)
- [Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples](https://mlsanity.com/paper/2010.10474/) (2020)
- [Density estimation in representation space to predict model uncertainty](https://mlsanity.com/paper/1908.07235/) (2019)
- [Inhibited Softmax for Uncertainty Estimation in Neural Networks](https://mlsanity.com/paper/1810.01861/) (2018)
- [It's always personal: Using Early Exits for Efficient On-Device CNN Personalisation](https://mlsanity.com/paper/2102.01393/) (2021)
- [Fast Video Classification via Adaptive Cascading of Deep Models](https://mlsanity.com/paper/1611.06453/) (2016)
- [Lightweight compression of neural network feature tensors for collaborative intelligence](https://mlsanity.com/paper/2105.06002/) (2021)
- [FastDeepIoT: Towards Understanding and Optimizing Neural Network Execution Time on Mobile and Embedded Devices](https://mlsanity.com/paper/1809.06970/) (2018)
- [Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles](https://mlsanity.com/paper/2007.00302/) (2020)
- [Packet-Loss-Tolerant Split Inference for Delay-Sensitive Deep Learning in Lossy Wireless Networks](https://mlsanity.com/paper/2104.13629/) (2021)

## Abstract (arXiv)

While existing strategies to execute deep learning-based classification on low-power platforms assume the models are trained on all classes of interest, this paper posits that adopting context-awareness i.e. narrowing down a classification task to the current deployment context consisting of only recent inference queries can substantially enhance performance in resource-constrained environments. We propose a new paradigm, CACTUS, for scalable and efficient context-aware classification where a micro-classifier recognizes a small set of classes relevant to the current context and, when context change happens (e.g., a new class comes into the scene), rapidly switches to another suitable micro-classifier. CACTUS features several innovations, including optimizing the training cost of context-aware classifiers, enabling on-the-fly context-aware switching between classifiers, and balancing context switching costs and performance gains via simple yet effective switching policies. We show that CACTUS achieves significant benefits in accuracy, latency, and compute budget across a range of datasets and IoT platforms.