mlsanityData to 7 Oct 2026

Activation Map Compression through Tensor Decomposition for Deep Learning

arXiv 2411.06346 ↗First posted 10 Nov 2024cs.LG cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Internet of Things and Deep Learning are synergetically and exponentially growing industrial fields with a massive call for their unification into a common framework called Edge AI. While on-device inference is a well-explored topic in recent research, backpropagation remains an open challenge due to its prohibitive computational and memory costs compared to the extreme resource constraints of embedded devices. Drawing on tensor decomposition research, we tackle the main bottleneck of backpropagation, namely the memory footprint of activation map storage. We investigate and compare the effects of activation compression using Singular Value Decomposition and its tensor variant, High-Order Singular Value Decomposition. The application of low-order decomposition results in considerable memory savings while preserving the features essential for learning, and also offers theoretical guarantees to convergence. Experimental results obtained on main-stream architectures and tasks demonstrate Pareto-superiority over other state-of-the-art solutions, in terms of the trade-off between generalization and memory footprint.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 57 references that are in the map. First posted on arXiv 10 Nov 2024.

Builds on

35 of 57 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2016Pyramid Scene Parsing Networkcited by 2,472
  11. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  12. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
23 more
  1. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  4. 2020Image Segmentation Using Deep Learning: A Surveycited by 659
  5. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  6. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 269
  7. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 262
  8. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
  9. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  10. 2022On-Device Training Under 256KB Memorycited by 158
  11. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  12. 2022Online Continual Learning for Embedded Devicescited by 48
  13. 2018Accelerating Convolutional Neural Networks via Activation Map Compressioncited by 34
  14. 2022Wavelet Feature Maps Compression for Image-to-Image CNNscited by 16
  15. 2021Convolutional Neural Network Compression through Generalized Kronecker Product Decompositioncited by 14
  16. 2023Efficient On-device Training via Gradient Filteringcited by 13
  17. 20220/1 Deep Neural Networks via Block Coordinate Descentcited by 10
  18. 2023On-Device Unsupervised Image Segmentationcited by 10
  19. 2023Efficient Low-rank Backpropagation for Vision Transformer Adaptationcited by 10
  20. 2023Towards On-device Learning on the Edge: Ways to Select Neurons to Update under a Budget Constraintcited by 6
  21. 2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Trainingcited by 5
  22. 2023Activation Compression of Graph Neural Networks using Block-wise Quantization with Improved Variance Minimizationcited by 4
  23. 2021Semi-tensor Product-based TensorDecomposition for Neural Network Compressioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learningcited by 5
  2. 2025LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learningcited by 2
  3. 2025Efficient Adaptation of Deep Neural Networks for Semantic Segmentation in Space Applicationscited by 1
  4. 2025Study of Training Dynamics for Memory-Constrained Fine-Tuningcited by 1
  5. 2026DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Trainingcited by 1
  6. 2025Efficient Resource-Constrained Training of Transformers via Subspace Optimizationcited by 0