mlsanityData to 14 Oct 2026

MACKO: Sparse Matrix-Vector Multiplication for Low Sparsity

arXiv 2511.13061 ↗First posted 17 Nov 2025cs.LG cs.AI cs.DCcited by 6

Not on the map. Why?

Abstract (arXiv)

Sparse Matrix-Vector Multiplication (SpMV) is a fundamental operation in the inference of sparse Large Language Models (LLMs). Because existing SpMV methods perform poorly under the low and unstructured sparsity (30-90%) commonly observed in pruned LLMs, unstructured pruning provided only limited memory reduction and speedup. We propose MACKO-SpMV, a GPU-optimized format and kernel co-designed to reduce storage overhead while preserving compatibility with the GPU's execution model. This enables efficient SpMV for unstructured sparsity without specialized hardware units (e.g., tensor cores) or format-specific precomputation. Empirical results show that at sparsity 50%, MACKO is the first approach with significant 1.5x memory reduction and 1.2-1.5x speedup over dense representation. Speedups over other SpMV baselines: 2.8-13.0x over cuSPARSE, 1.9-2.6x over Sputnik, and 2.2-2.5x over DASP. Applied to Llama2-7B pruned with Wanda to sparsity 50%, it delivers 1.5x memory reduction and 1.5x faster inference at fp16 precision. Thanks to MACKO, unstructured pruning at 50% sparsity is now justified in real-world LLM workloads.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 42 of its 65 references that are in the map. First posted on arXiv 17 Nov 2025.

Builds on

42 of 65 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2024Mixtral of Expertscited by 1,823
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
30 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  3. 2024Phi-4 Technical Reportcited by 711
  4. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  8. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  10. 20218-bit Optimizers via Block-wise Quantizationcited by 403
  11. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  12. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  13. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  14. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 189
  15. 2021Block Pruning For Faster Transformerscited by 177
  16. 2020Sparse GPU Kernels for Deep Learningcited by 141
  17. 2024QTIP: Quantization with Trellises and Incoherence Processingcited by 91
  18. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 84
  19. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  20. 2025Pretraining Large Language Models with NVFP4cited by 76
  21. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  22. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32
  23. 2024Benchmarking and Dissecting the Nvidia Hopper GPU Architecturecited by 24
  24. 2023VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Corescited by 13
  25. 2024Fast and Effective Weight Update for Pruned Large Language Modelscited by 11
  26. 2013A unified sparse matrix data format for efficient general sparse matrix-vector multiply on modern processors with wide SIMD unitscited by 6
  27. 2015CSR5: An Efficient Storage Format for Cross-Platform Sparse Matrix-Vector Multiplicationcited by 6
  28. 2024Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorizationcited by 6
  29. 2025The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMMcited by 5
  30. 2025Toward Efficient SpMV in Sparse LLMs via Block Extraction and Compressed Storagecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compressioncited by 2
  2. 2026Sparser, Faster, Lighter Transformer Language Modelscited by 0
  3. 2026LEAP: Learnable End-to-End Adaptive Pruning of Large Language Modelscited by 0
  4. 2026Post-Training Pruning for Diffusion Transformerscited by 0
  5. 2026Celty: SpMSpV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inferencecited by 0
  6. 2026The Sparsity Whisperercited by 0