mlsanityData to 14 Oct 2026

Language model compression with weighted low-rank factorization

arXiv 2207.00112 ↗First posted 30 Jun 2022cs.LG cs.AI cs.CLcited by 195

Not on the map. Why?

Cited by 20 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Factorizing a large matrix into small matrices is a popular strategy for model compression. Singular value decomposition (SVD) plays a vital role in this compression strategy, approximating a learned matrix with fewer parameters. However, SVD minimizes the squared error toward reconstructing the original matrix without gauging the importance of the parameters, potentially giving a larger reconstruction error for those who affect the task accuracy more. In other words, the optimization objective of SVD is not aligned with the trained model's task accuracy. We analyze this previously unexplored problem, make observations, and address it by introducing Fisher information to weigh the importance of parameters affecting the model prediction. This idea leads to our method: Fisher-Weighted SVD (FWSVD). Although the factorized matrices from our approach do not result in smaller reconstruction errors, we find that our resulting task accuracy is much closer to the original model's performance. We perform analysis with the transformer-based language models, showing our weighted SVD largely alleviates the mismatched optimization objectives and can maintain model performance with a higher compression rate. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies requiring expensive model pre-training. Moreover, the evaluation of compressing an already compact model shows our method can further reduce 9% to 30% parameters with an insignificant impact on task accuracy.

Key facts. Cited by 195 papers in the map (data through 14 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

25 of 36 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  11. 2018Neural Network Acceptability Judgmentscited by 958
  12. 2019Importance Estimation for Neural Network Pruningcited by 541
13 more
  1. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  2. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  3. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  4. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  5. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  6. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  7. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  8. 2021Group Fisher Pruning for Practical Network Compressioncited by 131
  9. 2019Fine-tune BERT for Extractive Summarizationcited by 122
  10. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
  11. 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
  12. 2018Online Embedding Compression for Text Classification using Low Rank Matrix Factorizationcited by 22
  13. 2022Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understandingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 21; 2024: 36; 2025: 69; 2026: 67.

195 papers.

Sort:

By area:

  1. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  2. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  3. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  4. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  5. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
  6. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  7. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  8. 2023Fast Inference of Mixture-of-Experts Language Models with Offloadingcited by 91
  9. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 72
  10. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 67
  11. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 64
  12. 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 59
38 more
  1. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  2. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  3. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 42
  4. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  5. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 35
  6. 2023Tensor Networks Meet Neural Networks: A Survey and Future Perspectivescited by 32
  7. 2022TrojViT: Trojan Insertion in Vision Transformerscited by 31
  8. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  9. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  10. 2023TrojText: Test-time Invisible Textual Trojan Insertioncited by 28
  11. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  12. 2025Wanda++: Pruning Large Language Models via Regional Gradientscited by 26
  13. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
  14. 2024Small Singular Values Matter: A Random Matrix Analysis of Transformer Modelscited by 21
  15. 2025SlimLLM: Accurate Structured Pruning for Large Language Modelscited by 18
  16. 2023TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decompositioncited by 17
  17. 2025You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learningcited by 17
  18. 2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibrationcited by 16
  19. 2025QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Modelscited by 16
  20. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  21. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 15
  22. 2022Numerical Optimizations for Weighted Low-rank Estimation on Language Modelcited by 14
  23. 2023COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Modelscited by 14
  24. 2024LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Modelscited by 12
  25. 2025Accelerating Diffusion Transformer via Increment-Calibrated Caching with Channel-Aware Singular Value Decompositioncited by 12
  26. 2026SAES-SVD: Self-Adaptive Suppression of Accumulated and Local Errors for SVD-based LLM Compressioncited by 12
  27. 2023Low-Rank Prune-And-Factorize for Language Model Compressioncited by 11
  28. 2026LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Modelscited by 11
  29. 2024Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimizationcited by 10
  30. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  31. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 10
  32. 2023Efficient Alternating Minimization with Applications to Weighted Low Rank Approximationcited by 9
  33. 2024Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inferencecited by 9
  34. 2024PC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillationcited by 9
  35. 2025FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Modelscited by 9
  36. 2023Transformer-Based Autonomous Driving Models and Deployment-Oriented Compression: A Surveycited by 8
  37. 2024KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Modelscited by 8
  38. 2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 8

50 most cited of 195 All 195 (JSON)