# Boosting Mobile CNN Inference through Semantic Memory

arXiv: https://arxiv.org/abs/2112.02644 · HTML: https://mlsanity.com/paper/2112.02644/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 63 references that are in the map.
- First posted on arXiv 5 Dec 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [SkipNet: Learning Dynamic Routing in Convolutional Networks](https://mlsanity.com/paper/1711.09485/) (2017)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Spatially Adaptive Computation Time for Residual Networks](https://mlsanity.com/paper/1612.02297/) (2016)
- [SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud](https://mlsanity.com/paper/2008.06402/) (2020)
- [PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning](https://mlsanity.com/paper/2001.00138/) (2020)
- [DeepCache: Principled Cache for Mobile Deep Vision](https://mlsanity.com/paper/1712.01670/) (2017)
- [Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascade](https://mlsanity.com/paper/1704.01344/) (2017)
- [A First Look at Deep Learning Apps on Smartphones](https://mlsanity.com/paper/1812.05448/) (2018)
- [Deep Representation Learning with Part Loss for Person Re-Identification](https://mlsanity.com/paper/1707.00798/) (2017)
- [GLAD: Global-Local-Alignment Descriptor for Pedestrian Retrieval](https://mlsanity.com/paper/1709.04329/) (2017)
- [EVA2: Exploiting Temporal Redundancy in Live Computer Vision](https://mlsanity.com/paper/1803.06312/) (2018)
- [CBinfer: Change-Based Inference for Convolutional Neural Networks on Video Data](https://mlsanity.com/paper/1704.04313/) (2017)

## Built on by (most cited first, 6 of 6)

- [LUT-NN: Empower Efficient Neural Network Inference with Centroid Learning and Table Lookup](https://mlsanity.com/paper/2302.03213/) (2023)
- [E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity](https://mlsanity.com/paper/2310.15929/) (2023)
- [LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Accelerator](https://mlsanity.com/paper/2501.10658/) (2025)
- [Accelerating End-Cloud Collaborative Inference via Near Bubble-free Pipeline Optimization](https://mlsanity.com/paper/2501.12388/) (2024)
- [A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques](https://mlsanity.com/paper/2302.08571/) (2023)
- [Dynamic neural network with memristive CIM and CAM for 2D and 3D vision](https://mlsanity.com/paper/2407.08990/) (2024)

## Abstract (arXiv)

Human brains are known to be capable of speeding up visual recognition of repeatedly presented objects through faster memory encoding and accessing procedures on activated neurons. For the first time, we borrow and distill such a capability into a semantic memory design, namely SMTM, to improve on-device CNN inference. SMTM employs a hierarchical memory architecture to leverage the long-tail distribution of objects of interest, and further incorporates several novel techniques to put it into effects: (1) it encodes high-dimensional feature maps into low-dimensional, semantic vectors for low-cost yet accurate cache and lookup; (2) it uses a novel metric in determining the exit timing considering different layers' inherent characteristics; (3) it adaptively adjusts the cache size and semantic vectors to fit the scene dynamics. SMTM is prototyped on commodity CNN engine and runs on both mobile CPU and GPU. Extensive experiments on large-scale datasets and models show that SMTM can significantly speed up the model inference over standard approach (up to 2X) and prior cache designs (up to 1.5X), with acceptable accuracy loss.