# Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping

arXiv: https://arxiv.org/abs/2308.07641 · HTML: https://mlsanity.com/paper/2308.07641/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 27 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Accelerating Very Deep Convolutional Networks for Classification and Detection](https://mlsanity.com/paper/1505.06798/) (2015)
- [Fast Algorithms for Convolutional Neural Networks](https://mlsanity.com/paper/1509.09308/) (2015)
- [AdderNet: Do We Really Need Multiplications in Deep Learning?](https://mlsanity.com/paper/1912.13200/) (2019)
- [Prune Once for All: Sparse Pre-Trained Language Models](https://mlsanity.com/paper/2111.05754/) (2021)
- [An Overview of Neural Network Compression](https://mlsanity.com/paper/2006.03669/) (2020)
- [ShiftAddNet: A Hardware-Inspired Deep Network](https://mlsanity.com/paper/2010.12785/) (2020)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [Towards Compact CNNs via Collaborative Compression](https://mlsanity.com/paper/2105.11228/) (2021)
- [Seq2Biseq: Bidirectional Output-wise Recurrent Neural Networks for Sequence Modelling](https://mlsanity.com/paper/1904.04733/) (2019)
- [Compact Model Training by Low-Rank Projection with Energy Transfer](https://mlsanity.com/paper/2204.05566/) (2022)

## Built on by (most cited first, 3 of 3)

- [A Survey on Transformer Compression](https://mlsanity.com/paper/2402.05964/) (2024)
- [ESPACE: Dimensionality Reduction of Activations for Model Compression](https://mlsanity.com/paper/2410.05437/) (2024)
- [Optimizing Singular Spectrum for Large Language Model Compression](https://mlsanity.com/paper/2502.15092/) (2025)

## Abstract (arXiv)

We present a simple yet novel parameterized form of linear mapping to achieves remarkable network compression performance: a pseudo SVD called Ternary SVD (TSVD). Unlike vanilla SVD, TSVD limits the U and V matrices in SVD to ternary matrices form in ± 1, 0. This means that instead of using the expensive multiplication instructions, TSVD only requires addition instructions when computing U(·) and V(·). We provide direct and training transition algorithms for TSVD like Post Training Quantization and Quantization Aware Training respectively. Additionally, we analyze the convergence of the direct transition algorithms in theory. In experiments, we demonstrate that TSVD can achieve state-of-the-art network compression performance in various types of networks and tasks, including current baseline models such as ConvNext, Swim, BERT, and large language model like OPT.