mlsanityData to 14 Oct 2026

CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning

arXiv 2509.22075 ↗First posted 26 Sep 2025cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Post-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose CoSpaDi, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. CoSpaDi factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across Llama and Qwen models, CoSpaDi improves accuracy--compression and perplexity--compression trade-offs over SVD-based and structured pruning baselines at 20--40% compression ratios, while naturally supporting sparse--dense computation and post-training quantization of sparse coefficients.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 63 of its 97 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

63 of 97 references.

References · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  13. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  14. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  15. 2023PaLM 2 Technical Reportcited by 1,162
  16. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  17. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  18. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  19. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  20. 2016Sequence-Level Knowledge Distillationcited by 801
  21. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  22. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  23. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  24. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  25. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  26. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  27. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  28. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  29. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  30. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  31. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  32. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  33. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  34. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  35. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
  36. 2021Accelerating Sparse Deep Neural Networkscited by 223
  37. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  38. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 201

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Modelscited by 3
  2. 2026ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compressioncited by 2
  3. 2026COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compressioncited by 1
  4. 2026Motion-Compensated Weight Compressioncited by 0
  5. 2026Output-Space Allocation Costs for Calibration-Guided LLM Compression: An Empirical Studycited by 0
  6. 2026GeoPair: Geometry-Preserving Cross-Layer Factorization for Training-Free Transformer Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs
  2. 2024SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
  3. 2026FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression
  4. 2024EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation
  5. 2026SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
  6. 2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression
  7. 2024Compression for Better: A General and Stable Lossless Compression Framework
  8. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simple