# Accelerating Convolutional Neural Networks via Activation Map Compression

arXiv: https://arxiv.org/abs/1812.04056 · HTML: https://mlsanity.com/paper/1812.04056/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2018.

## Built on by (most cited first, 30 of 34)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models](https://mlsanity.com/paper/2402.13516/) (2024)
- [NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks](https://mlsanity.com/paper/2206.09117/) (2022)
- [Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees](https://mlsanity.com/paper/2206.01299/) (2022)
- [Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation](https://mlsanity.com/paper/2503.01776/) (2025)
- [DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device](https://mlsanity.com/paper/2104.09949/) (2021)
- [Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion](https://mlsanity.com/paper/2310.04361/) (2023)
- [Emergent representations in networks trained with the Forward-Forward algorithm](https://mlsanity.com/paper/2305.18353/) (2023)
- [Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity](https://mlsanity.com/paper/2309.06626/) (2023)
- [Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training](https://mlsanity.com/paper/2411.13055/) (2024)
- [Aggregating Capacity in FL through Successive Layer Training for Computationally-Constrained Devices](https://mlsanity.com/paper/2305.17005/) (2023)
- [The Impact of Uniform Inputs on Activation Sparsity and Energy-Latency Attacks in Computer Vision](https://mlsanity.com/paper/2403.18587/) (2024)
- [Event-based Optical Flow on Neuromorphic Processor: ANN vs. SNN Comparison based on Activation Sparsification](https://mlsanity.com/paper/2407.20421/) (2024)
- [Survey: Exploiting Data Redundancy for Optimization of Deep Learning](https://mlsanity.com/paper/2208.13363/) (2022)
- [Activation Map Compression through Tensor Decomposition for Deep Learning](https://mlsanity.com/paper/2411.06346/) (2024)
- [Deep Neural Network Initialization with Sparsity Inducing Activations](https://mlsanity.com/paper/2402.16184/) (2024)
- [Optimizing the Consumption of Spiking Neural Networks with Activity Regularization](https://mlsanity.com/paper/2204.01460/) (2022)
- [DIVISION: Memory Efficient Training via Dual Activation Precision](https://mlsanity.com/paper/2208.04187/) (2022)
- [SPADE: Sparse Pillar-based 3D Object Detection Accelerator for Autonomous Driving](https://mlsanity.com/paper/2305.07522/) (2023)
- [TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Network](https://mlsanity.com/paper/2506.01352/) (2025)
- [High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks](https://mlsanity.com/paper/2104.08314/) (2021)
- [Sparse Convolutional Recurrent Learning for Efficient Event-based Neuromorphic Object Detection](https://mlsanity.com/paper/2506.13440/) (2025)
- [Training for temporal sparsity in deep neural networks, application in video processing](https://mlsanity.com/paper/2107.07305/) (2021)
- [Improved Projection Learning for Lower Dimensional Feature Maps](https://mlsanity.com/paper/2210.15170/) (2022)
- [An Efficient Split Fine-tuning Framework for Edge and Cloud Collaborative Learning](https://mlsanity.com/paper/2211.16703/) (2022)
- [Bandwidth-efficient Inference for Neural Image Compression](https://mlsanity.com/paper/2309.02855/) (2023)
- [FlatENN: Train Flat for Enhanced Fault Tolerance of Quantized Deep Neural Networks](https://mlsanity.com/paper/2301.00675/) (2022)
- [Dual sparse training framework: inducing activation map sparsity via Transformed ℓ1 regularization](https://mlsanity.com/paper/2405.19652/) (2024)

## Abstract (arXiv)

The deep learning revolution brought us an extensive array of neural network architectures that achieve state-of-the-art performance in a wide variety of Computer Vision tasks including among others, classification, detection and segmentation. In parallel, we have also been observing an unprecedented demand in computational and memory requirements, rendering the efficient use of neural networks in low-powered devices virtually unattainable. Towards this end, we propose a three-stage compression and acceleration pipeline that sparsifies, quantizes and entropy encodes activation maps of Convolutional Neural Networks. Sparsification increases the representational power of activation maps leading to both acceleration of inference and higher model accuracy. Inception-V3 and MobileNet-V1 can be accelerated by as much as 1.6× with an increase in accuracy of 0.38% and 0.54% on the ImageNet and CIFAR-10 datasets respectively. Quantizing and entropy coding the sparser activation maps lead to higher compression over the baseline, reducing the memory cost of the network execution. Inception-V3 and MobileNet-V1 activation maps, quantized to 16 bits, are compressed by as much as 6× with an increase in accuracy of 0.36% and 0.55% respectively.