mlsanityData to 14 Oct 2026

SparseLoRA: Accelerating LLM Fine-Tuning with Contextual Sparsity

arXiv 2506.16500 ↗First posted 19 Jun 2025cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Fine-tuning LLMs is both computationally and memory-intensive. While parameter-efficient fine-tuning methods, such as QLoRA and DoRA, reduce the number of trainable parameters and lower memory usage, they do not decrease computational cost. In some cases, they may even slow down fine-tuning. In this paper, we introduce SparseLoRA, a method that accelerates LLM fine-tuning through contextual sparsity. We propose a lightweight, training-free SVD sparsity estimator that dynamically selects a sparse subset of weights for loss and gradient computation. Also, we systematically analyze and address sensitivity across layers, tokens, and training steps. Our experimental results show that SparseLoRA reduces computational cost by up to 2.2 times and a measured speedup of up to 1.6 times while maintaining accuracy across various downstream tasks, including commonsense and arithmetic reasoning, code generation, and instruction following.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 59 of its 69 references that are in the map. First posted on arXiv 19 Jun 2025.

Builds on

59 of 69 references.

References · 59

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  9. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  11. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  13. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  14. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  15. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  16. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  17. 2024Capabilities of Gemini Models in Medicinecited by 237
  18. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  19. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  20. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 208
  21. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  22. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  23. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
  24. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  25. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 114
  26. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  27. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  28. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  29. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 77
  30. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 67
  31. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  32. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
  33. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  34. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 38
  35. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 38
  36. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  37. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 28
  38. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 7.

9 papers.

  1. 2025Dynamic Rank Adjustment for Accurate and Efficient Neural Network Trainingcited by 1
  2. 2026FourTune: Towards Fully 4-Bit Efficient Post-Training for Diffusion Modelscited by 1
  3. 2025Selective LoRA for Visual Tokens and Attention Headscited by 0
  4. 2026SVD Contextual Sparsity Predictors for Fast LLM Inferencecited by 0
  5. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0
  6. 2026Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMscited by 0
  7. 2026Source-Prior-Driven Selective Adaptation for Efficient Diffusion Model Finetuningcited by 0
  8. 2026AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuningcited by 0
  9. 2026Fine-Tuning of Transformer models with Framescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Models
  2. 2026How Low Can You Go: Fine-Tuning LLMs with Bayesian Optimization (LlMBO)
  3. 2025Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
  4. 2024Scaling Sparse Fine-Tuning to Large Language Models
  5. 2025LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
  6. 2025An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruning
  7. 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
  8. 2025Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients