mlsanityData to 14 Oct 2026

LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation

arXiv 2306.11222 ↗First posted 20 Jun 2023cs.LG cs.CLcited by 104

Not on the map. Why?

Abstract (arXiv)

Transformer models have achieved remarkable results in various natural language tasks, but they are often prohibitively large, requiring massive memories and computational resources. To reduce the size and complexity of these models, we propose LoSparse (Low-Rank and Sparse approximation), a novel model compression technique that approximates a weight matrix by the sum of a low-rank matrix and a sparse matrix. Our method combines the advantages of both low-rank approximations and pruning, while avoiding their limitations. Low-rank approximation compresses the coherent and expressive parts in neurons, while pruning removes the incoherent and non-expressive parts in neurons. Pruning enhances the diversity of low-rank approximations, and low-rank approximation prevents pruning from losing too many expressive neurons. We evaluate our method on natural language understanding, question answering, and natural language generation tasks. We show that it significantly outperforms existing compression methods.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

33 of 54 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
21 more
  1. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  2. 2015Teaching Machines to Read and Comprehendcited by 1,049
  3. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  4. 2018Neural Network Acceptability Judgmentscited by 958
  5. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  6. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  7. 2019Importance Estimation for Neural Network Pruningcited by 541
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  9. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  10. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  11. 2022Language model compression with weighted low-rank factorizationcited by 195
  12. 2021Block Pruning For Faster Transformerscited by 177
  13. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  14. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 113
  15. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  16. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  17. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  18. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  19. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 20
  20. 2021DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Modelscited by 8
  21. 2021Low-Rank+Sparse Tensor Compression for Neural Networkscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 38; 2025: 36; 2026: 20.

104 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  3. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  4. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  6. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  7. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  8. 2023Efficient Large Language Models: A Surveycited by 186
  9. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  10. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 169
  11. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  12. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 108
38 more
  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  2. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  3. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  4. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  5. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 72
  6. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  7. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  8. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  9. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  10. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  11. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  12. 2024BlockPruner: Fine-grained Pruning for Large Language Modelscited by 36
  13. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33
  14. 2024Deep Learning for Cross-Domain Data Fusion in Urban Computing: Taxonomy, Advances, and Outlookcited by 33
  15. 2025Delta Decompression for MoE-based LLMs Compressioncited by 33
  16. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  17. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 25
  18. 2024SparseLLM: Towards Global Pruning for Pre-trained Language Modelscited by 23
  19. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forwardcited by 21
  20. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 21
  21. 2024RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuningcited by 21
  22. 2024SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMscited by 20
  23. 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 20
  24. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 16
  25. 2024AFLoRA: Adaptive Freezing of Low Rank Adaptation in Parameter Efficient Fine-Tuning of Large Modelscited by 15
  26. 2025SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compressioncited by 14
  27. 2024CliqueParcel: An Approach For Batching LLM Prompts That Jointly Optimizes Efficiency And Faithfulnesscited by 12
  28. 2024LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Modelscited by 12
  29. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compressioncited by 12
  30. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
  31. 2025Identifying Sensitive Weights via Post-quantization Integralcited by 11
  32. 2025FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRAcited by 11
  33. 2024Adaptive Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimizationcited by 10
  34. 2025Dynamic Low-Rank Sparse Adaptation for Large Language Modelscited by 10
  35. 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
  36. 2025Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretabilitycited by 7
  37. 2025LatentLLM: Attention-Aware Joint Tensor Compressioncited by 7
  38. 2026Model Merging in the Essential Subspacecited by 7

50 most cited of 104 All 104 (JSON)