Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity
Not on the map. Why?
Abstract (arXiv)
Fully finetuning foundation language models (LMs) with billions of parameters is often impractical due to high computational costs, memory requirements, and the risk of overfitting. Although methods like low-rank adapters help address these challenges by adding small trainable modules to the frozen LM, they also increase memory usage and do not reduce inference latency. We uncover an intriguing phenomenon: sparsifying specific model rows and columns enables efficient task adaptation without requiring weight tuning. We propose a scheme for effective finetuning via sparsification using training stochastic gates, which requires minimal trainable parameters, reduces inference time, and removes 20--40% of model parameters without significant accuracy loss. Empirical results show it outperforms recent finetuning baselines in efficiency and performance. Additionally, we provide theoretical guarantees for the convergence of this stochastic gating process, and show that our method admits a simpler and better-conditioned optimization landscape compared to LoRA. Our results highlight sparsity as a compelling mechanism for task-specific adaptation in LMs.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 50 of its 60 references that are in the map. First posted on arXiv 9 Feb 2026.
Builds on
50 of 60 references.
References · 50
- 2017Attention Is All You Needcited by 59,932
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
38 more
- 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
- 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
- 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
- 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
- 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
- 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
- 2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
- 2024Adam-mini: Use Fewer Learning Rates To Gain Morecited by 134
- 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
- 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
- 2024Flora: Low-Rank Adapters Are Secretly Gradient Compressorscited by 103
- 2018Implicit Reparameterization Gradientscited by 84
- 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
- 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
- 2024LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameterscited by 73
- 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
- 2024MaskLLM: Learnable Semi-Structured Sparsity for Large Language Modelscited by 49
- 2024Low-Rank Quantization-Aware Training for LLMscited by 42
- 2024Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?cited by 39
- 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
- 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
- 2017Reducing Reparameterization Gradient Variancecited by 27
- 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
- 2024VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projectionscited by 18
- 2024Knowledge Editing in Language Models via Adapted Direct Preference Optimizationcited by 17
- 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 16
- 2025SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Trainingcited by 16
- 2024NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Modelscited by 13
- 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
- 2025Parameter-Efficient Fine-Tuning of Large Language Models via Deconvolution in Subspacecited by 11
- 2021Support Recovery with Stochastic Gates: Theory and Application for Linear Modelscited by 8
- 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8
- 2023Multi-modal Differentiable Unsupervised Feature Selectioncited by 5
- 2025Hybrid Autoencoders for Tabular Data: Leveraging Model-Based Augmentation in Low-Label Settingscited by 5
- 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 3
- 2022Learning k-Level Structured Sparse Neural Networks Using Group Envelope Regularizationcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.