mlsanityData to 7 Oct 2026

Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

arXiv 2607.09287 ↗First posted 10 Jul 2026cs.LG cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 24 references that are in the map. First posted on arXiv 10 Jul 2026.

Builds on

21 of 24 references.

References · 21

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  6. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  7. 2016Solving General Arithmetic Word Problemscited by 353
  8. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  9. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  10. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  11. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  12. 2024Scaling Sparse Fine-Tuning to Large Language Modelscited by 25
9 more
  1. 2023Sparse is Enough in Fine-tuning Pre-trained Large Language Modelscited by 21
  2. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  3. 2024Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Modelcited by 8
  4. 2024GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMscited by 8
  5. 2024RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updatescited by 8
  6. 2024Sparsity-Accelerated Training for Large Language Modelscited by 5
  7. 2024Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Modelscited by 3
  8. 2025An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruningcited by 1
  9. 2025GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameterscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026What Should We Freeze? Guarded Freezing: Connectivity Shapes the Fine-Tuning of Pretrained Modelscited by 0