# Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

arXiv: https://arxiv.org/abs/2607.09287 · HTML: https://mlsanity.com/paper/2607.09287/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 10 Jul 2026.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Solving General Arithmetic Word Problems](https://mlsanity.com/paper/1608.01413/) (2016)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [Parameter-Efficient Fine-Tuning without Introducing New Latency](https://mlsanity.com/paper/2305.16742/) (2023)
- [RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation](https://mlsanity.com/paper/2401.04679/) (2024)
- [SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining](https://mlsanity.com/paper/2406.02214/) (2024)
- [Scaling Sparse Fine-Tuning to Large Language Models](https://mlsanity.com/paper/2401.16405/) (2024)
- [Sparse is Enough in Fine-tuning Pre-trained Large Language Models](https://mlsanity.com/paper/2312.11875/) (2023)
- [S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity](https://mlsanity.com/paper/2412.06289/) (2024)
- [Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model](https://mlsanity.com/paper/2403.11621/) (2024)
- [GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs](https://mlsanity.com/paper/2408.15300/) (2024)
- [RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates](https://mlsanity.com/paper/2410.10075/) (2024)
- [Sparsity-Accelerated Training for Large Language Models](https://mlsanity.com/paper/2406.01392/) (2024)
- [Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models](https://mlsanity.com/paper/2412.13488/) (2024)
- [An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruning](https://mlsanity.com/paper/2502.11439/) (2025)
- [GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters](https://mlsanity.com/paper/2510.19778/) (2025)

## Built on by (most cited first, 1 of 1)

- [What Should We Freeze? Guarded Freezing: Connectivity Shapes the Fine-Tuning of Pretrained Models](https://mlsanity.com/paper/2609.32124/) (2026)

## Abstract (arXiv)

Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.