mlsanityData to 7 Oct 2026

Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization

arXiv 2409.18850 ↗First posted 27 Sep 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Neural networks are often challenging to work with due to their large size and complexity. To address this, various methods aim to reduce model size by sparsifying or decomposing weight matrices, such as magnitude pruning and low-rank or block-diagonal factorization. In this work, we present Double Sparse Factorization (DSF), where we factorize each weight matrix into two sparse matrices. Although solving this problem exactly is computationally infeasible, we propose an efficient heuristic based on alternating minimization via ADMM that achieves state-of-the-art results, enabling unprecedented sparsification of neural networks. For instance, in a one-shot pruning setting, our method can reduce the size of the LLaMA2-13B model by 50% while maintaining better performance than the dense LLaMA2-7B model. We also compare favorably with Optimal Brain Compression, the state-of-the-art layer-wise pruning approach for convolutional neural networks. Furthermore, accuracy improvements of our method persist even after further model fine-tuning. Code available at: https://github.com/usamec/double_sparse.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 52 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

34 of 52 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
22 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  6. 2018Rethinking the Value of Network Pruningcited by 595
  7. 2020What is the State of Neural Network Pruning?cited by 578
  8. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  9. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  10. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  11. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  12. 2020Sparse GPU Kernels for Deep Learningcited by 138
  13. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  14. 2019QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutionscited by 93
  15. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  16. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 73
  17. 2024PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compressioncited by 59
  18. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  19. 2024Fast and Effective Weight Update for Pruned Large Language Modelscited by 11
  20. 2022Cyclical Pruning for Sparse Neural Networkscited by 6
  21. 2015Flexible Multi-layer Sparse Approximations of Matrices and Applicationscited by 5
  22. 2020Nanopore Base Calling on the Edgecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025MACKO: Sparse Matrix-Vector Multiplication for Low Sparsitycited by 6
  2. 2025Addition is almost all you need: Compressing large language models with double binary factorizationcited by 3
  3. 2025ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorizationcited by 2
  4. 2026Sparse Weight Decomposition for Efficient Circuit Extractioncited by 1
  5. 2025Optimal Singular Damage: Efficient LLM Inference in Low Storage Regimescited by 0
  6. 2026Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuningcited by 0