# Activation Map Compression through Tensor Decomposition for Deep Learning

arXiv: https://arxiv.org/abs/2411.06346 · HTML: https://mlsanity.com/paper/2411.06346/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 57 references that are in the map.
- First posted on arXiv 10 Nov 2024.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [Accelerating Very Deep Convolutional Networks for Classification and Detection](https://mlsanity.com/paper/1505.06798/) (2015)
- [Online Continual Learning for Embedded Devices](https://mlsanity.com/paper/2203.10681/) (2022)
- [Accelerating Convolutional Neural Networks via Activation Map Compression](https://mlsanity.com/paper/1812.04056/) (2018)
- [Wavelet Feature Maps Compression for Image-to-Image CNNs](https://mlsanity.com/paper/2205.12268/) (2022)
- [Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition](https://mlsanity.com/paper/2109.14710/) (2021)
- [Efficient On-device Training via Gradient Filtering](https://mlsanity.com/paper/2301.00330/) (2023)
- [0/1 Deep Neural Networks via Block Coordinate Descent](https://mlsanity.com/paper/2206.09379/) (2022)
- [On-Device Unsupervised Image Segmentation](https://mlsanity.com/paper/2303.12753/) (2023)

## Built on by (most cited first, 6 of 6)

- [Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning](https://mlsanity.com/paper/2505.05086/) (2025)
- [LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning](https://mlsanity.com/paper/2509.21617/) (2025)
- [Efficient Adaptation of Deep Neural Networks for Semantic Segmentation in Space Applications](https://mlsanity.com/paper/2504.15991/) (2025)
- [Study of Training Dynamics for Memory-Constrained Fine-Tuning](https://mlsanity.com/paper/2510.19675/) (2025)
- [DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training](https://mlsanity.com/paper/2606.28932/) (2026)
- [Efficient Resource-Constrained Training of Transformers via Subspace Optimization](https://mlsanity.com/paper/2510.09160/) (2025)

## Abstract (arXiv)

Internet of Things and Deep Learning are synergetically and exponentially growing industrial fields with a massive call for their unification into a common framework called Edge AI. While on-device inference is a well-explored topic in recent research, backpropagation remains an open challenge due to its prohibitive computational and memory costs compared to the extreme resource constraints of embedded devices. Drawing on tensor decomposition research, we tackle the main bottleneck of backpropagation, namely the memory footprint of activation map storage. We investigate and compare the effects of activation compression using Singular Value Decomposition and its tensor variant, High-Order Singular Value Decomposition. The application of low-order decomposition results in considerable memory savings while preserving the features essential for learning, and also offers theoretical guarantees to convergence. Experimental results obtained on main-stream architectures and tasks demonstrate Pareto-superiority over other state-of-the-art solutions, in terms of the trade-off between generalization and memory footprint.