mlsanityData to 7 Oct 2026

Semi-tensor Product-based TensorDecomposition for Neural Network Compression

arXiv 2109.15200 ↗First posted 30 Sep 2021cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The existing tensor networks adopt conventional matrix product for connection. The classical matrix product requires strict dimensionality consistency between factors, which can result in redundancy in data representation. In this paper, the semi-tensor product is used to generalize classical matrix product-based mode product to semi-tensor mode product. As it permits the connection of two factors with different dimensionality, more flexible and compact tensor decompositions can be obtained with smaller sizes of factors. Tucker decomposition, Tensor Train (TT) and Tensor Ring (TR) are common decomposition for low rank compression of deep neural networks. The semi-tensor product is applied to these tensor decompositions to obtained their generalized versions, i.e., semi-tensor Tucker decomposition (STTu), semi-tensor train(STT) and semi-tensor ring (STR). Experimental results show the STTu, STT and STR achieve higher compression factors than the conventional tensor decompositions with the same accuracy but less training times in ResNet and WideResNetcompression. With 2% accuracy degradation, the TT-RN (rank = 14) and the TR-WRN (rank = 16) only obtain 3 times and99t times compression factors while the STT-RN (rank = 14) and the STR-WRN (rank = 16) achieve 9 times and 179 times compression factors, respectively.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 30 Sep 2021.

Builds on

35 of 50 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  9. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  10. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  11. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  12. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
23 more
  1. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  2. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  3. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  4. 2015Tensorizing Neural Networkscited by 246
  5. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  6. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 161
  7. 2016Tensor Ring Decompositioncited by 125
  8. 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 121
  9. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  10. 2015A Primer on Neural Network Models for Natural Language Processingcited by 75
  11. 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
  12. 2016Ultimate tensorization: compressing convolutional and FC layers alikecited by 55
  13. 2018Wide Compression: Tensor Ring Netscited by 50
  14. 2017Efficient Low Rank Tensor Ring Completioncited by 45
  15. 2017Tensor Regression Networkscited by 44
  16. 2017Learning Compact Recurrent Neural Networks with Block-Term Tensor Decompositioncited by 33
  17. 2019T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensorcited by 22
  18. 2018Network Decoupling: From Regular to Depthwise Separable Convolutionscited by 17
  19. 2020Hybrid Tensor Decomposition in Neural Network Compressioncited by 15
  20. 2020Block-term Tensor Neural Networkscited by 13
  21. 2018Tensor Decomposition for Compressing Recurrent Neural Networkcited by 10
  22. 2020Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decompositioncited by 9
  23. 2017Tensor Regression Networks with various Low-Rank Tensor Approximationscited by 8

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 1.

3 papers.

  1. 2024Activation Map Compression through Tensor Decomposition for Deep Learningcited by 6
  2. 2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Trainingcited by 5
  3. 2025Efficient Resource-Constrained Training of Transformers via Subspace Optimizationcited by 0