mlsanityData to 7 Oct 2026

Tensor Decomposition for Model Reduction in Neural Networks: A Review

arXiv 2304.13539 ↗First posted 26 Apr 2023cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Modern neural networks have revolutionized the fields of computer vision (CV) and Natural Language Processing (NLP). They are widely used for solving complex CV tasks and NLP tasks such as image classification, image generation, and machine translation. Most state-of-the-art neural networks are over-parameterized and require a high computational cost. One straightforward solution is to replace the layers of the networks with their low-rank tensor approximations using different tensor decomposition methods. This paper reviews six tensor decomposition methods and illustrates their ability to compress model parameters of convolutional neural networks (CNNs), recurrent neural networks (RNNs) and Transformers. The accuracy of some compressed models can be higher than the original versions. Evaluations indicate that tensor decompositions can achieve significant reductions in model size, run-time and energy consumption, and are well suited for implementing neural networks on edge devices.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 50 of its 88 references that are in the map. First posted on arXiv 26 Apr 2023.

Builds on

50 of 88 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
38 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2016Language Modeling with Gated Convolutional Networkscited by 688
  4. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  5. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  6. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  7. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  8. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 287
  9. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  10. 2015Tensorizing Neural Networkscited by 246
  11. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  12. 2013Predicting Parameters in Deep Learningcited by 223
  13. 2016Tensor Decomposition for Signal Processing and Machine Learningcited by 223
  14. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  15. 2016Exploring the Limits of Language Modelingcited by 172
  16. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  17. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
  18. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
  19. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  20. 2016Tensor Ring Decompositioncited by 125
  21. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
  22. 2019A Tensorized Transformer for Language Modelingcited by 85
  23. 2016Edinburgh Neural Machine Translation Systems for WMT 16cited by 76
  24. 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
  25. 2017Tensor-Train Recurrent Neural Networks for Video Classificationcited by 59
  26. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 55
  27. 2015Video Paragraph Captioning Using Hierarchical Recurrent Neural Networkscited by 54
  28. 2014MatConvNet - Convolutional Neural Networks for MATLABcited by 36
  29. 2018Compressing Recurrent Neural Networks with Tensor Ring for Action Recognitioncited by 34
  30. 2017Learning Compact Recurrent Neural Networks with Block-Term Tensor Decompositioncited by 33
  31. 2017CP-decomposition with Tensor Power Method for Convolutional Neural Networks Compressioncited by 30
  32. 2019Tensorized Embedding Layers for Efficient Model Compressioncited by 27
  33. 2015PerforatedCNNs: Acceleration through Elimination of Redundant Convolutionscited by 17
  34. 2020PERMDNN: Efficient Compressed DNN Architecture with Permuted Diagonal Matricescited by 12
  35. 2020Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decompositioncited by 9
  36. 2022TDC: Towards Extremely Efficient CNNs on GPUs via Hardware-Aware Tucker Decompositioncited by 5
  37. 2019Matrix and tensor decompositions for training binary neural networkscited by 2
  38. 2022Algorithm and Hardware Co-Design of Energy-Efficient LSTM Networks for Video Recognition with Hierarchical Tucker Tensor Decompositioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 9; 2026: 2.

14 papers.

Sort:

By area:

  1. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  2. 2023Tensor Decomposition Based Attention Module for Spiking Neural Networkscited by 8
  3. 2024Transformers As Approximations of Solomonoff Inductioncited by 6
  4. 2025Theoretical Guarantees for Low-Rank Compression of Deep Neural Networkscited by 3
  5. 2025Towards Quantum Tensor Decomposition in Biomedical Applicationscited by 2
  6. 2025Low-Rank Tensor Decompositions for the Theory of Neural Networkscited by 1
  7. 2025AI Compute Architecture and Evolution Trendscited by 1
  8. 2026MixQuant: Adaptive Mixed-Precision Quantization for Large Language Modelscited by 1
  9. 2023Lightweight Attribute Localizing Models for Pedestrian Attribute Recognitioncited by 0
  10. 2025Intelligently Augmented Contrastive Tensor Factorization: Empowering Multi-dimensional Time Series Classification in Low-Data Environmentscited by 0
  11. 2025ReLATE: Accelerating Tensor Decomposition via Safe and Efficient Learning of Sparse Encodingscited by 0
  12. 2025Data Understanding Survey: Pursuing Improved Dataset Characterization Via Tensor-based Methodscited by 0
2 more
  1. 2025A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agentscited by 0
  2. 2026TT-SEAL: TTD-Aware Selective Encryption for Adversarially-Robust and Low-Latency Edge AIcited by 0