mlsanityData to 7 Oct 2026

Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping

arXiv 2308.07641 ↗First posted 15 Aug 2023cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

We present a simple yet novel parameterized form of linear mapping to achieves remarkable network compression performance: a pseudo SVD called Ternary SVD (TSVD). Unlike vanilla SVD, TSVD limits the U and V matrices in SVD to ternary matrices form in ± 1, 0. This means that instead of using the expensive multiplication instructions, TSVD only requires addition instructions when computing U(·) and V(·). We provide direct and training transition algorithms for TSVD like Post Training Quantization and Quantization Aware Training respectively. Additionally, we analyze the convergence of the direct transition algorithms in theory. In experiments, we demonstrate that TSVD can achieve state-of-the-art network compression performance in various types of networks and tasks, including current baseline models such as ConvNext, Swim, BERT, and large language model like OPT.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 24 of its 27 references that are in the map. First posted on arXiv 15 Aug 2023.

Builds on

24 of 27 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
12 more
  1. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  2. 2021Accelerating Sparse Deep Neural Networkscited by 213
  3. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  4. 2015Fast Algorithms for Convolutional Neural Networkscited by 102
  5. 2019AdderNet: Do We Really Need Multiplications in Deep Learning?cited by 82
  6. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  7. 2020An Overview of Neural Network Compressioncited by 59
  8. 2020ShiftAddNet: A Hardware-Inspired Deep Networkcited by 52
  9. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 48
  10. 2021Towards Compact CNNs via Collaborative Compressioncited by 36
  11. 2019Seq2Biseq: Bidirectional Output-wise Recurrent Neural Networks for Sequence Modellingcited by 4
  12. 2022Compact Model Training by Low-Rank Projection with Energy Transfercited by 3

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024A Survey on Transformer Compressioncited by 44
  2. 2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
  3. 2025Optimizing Singular Spectrum for Large Language Model Compressioncited by 4