mlsanityData to 14 Oct 2026

Efficient Model Editing with Task-Localized Sparse Fine-tuning

arXiv 2504.02620 ↗First posted 3 Apr 2025cs.LG cs.AI cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Task arithmetic has emerged as a promising approach for editing models by representing task-specific knowledge as composable task vectors. However, existing methods rely on network linearization to derive task vectors, leading to computational bottlenecks during training and inference. Moreover, linearization alone does not ensure weight disentanglement, the key property that enables conflict-free composition of task vectors. To address this, we propose TaLoS which allows to build sparse task vectors with minimal interference without requiring explicit linearization and sharing information across tasks. We find that pre-trained models contain a subset of parameters with consistently low gradient sensitivity across tasks, and that sparsely updating only these parameters allows for promoting weight disentanglement during fine-tuning. Our experiments prove that TaLoS improves training and inference efficiency while outperforming current methods in task addition and negation. By enabling modular parameter editing, our approach fosters practical deployment of adaptable foundation models in real-world applications.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 58 of its 77 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

58 of 77 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
38 more
  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  2. 2022Editing Models with Task Arithmeticcited by 1,120
  3. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  5. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  6. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  7. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  8. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  9. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
  10. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  11. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  12. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  13. 2020What is the State of Neural Network Pruning?cited by 583
  14. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  15. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  16. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  17. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  18. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  19. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  20. 2022The Forward-Forward Algorithm: Some Preliminary Investigationscited by 266
  21. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  22. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 252
  23. 2020Supermasks in Superpositioncited by 227
  24. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 221
  25. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 217
  26. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  27. 2022Patching open-vocabulary models by interpolating weightscited by 190
  28. 2020Training independent subnetworks for robust predictioncited by 177
  29. 2022A Fast Post-Training Pruning Framework for Transformerscited by 162
  30. 2013Revisiting Natural Gradient for Deep Networkscited by 159
  31. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  32. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  33. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 133
  34. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 131
  35. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 121
  36. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  37. 2019QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questionscited by 89
  38. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 65

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 6.

16 papers.

Sort:

By area:

  1. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 19
  2. 2025UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Modelscited by 9
  3. 2025Exploring Nonlinear Pathway in Parameter Space for Machine Unlearningcited by 4
  4. 2025From Memorization to Parameter Interference: How Overtraining Experts Harms Model Mergingcited by 4
  5. 2026Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvaturecited by 3
  6. 2025Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignmentcited by 1
  7. 2025What Should Feature Distillation Transfer in LLMs? A Task-Tangent Geometry Viewcited by 1
  8. 2026Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmeticcited by 1
  9. 2026Training-free Task Classification for Multi-Task Model Mergingcited by 1
  10. 2025Efficient Multi-Source Knowledge Transfer by Model Mergingcited by 0
  11. 2025Variational Task Vector Compositioncited by 0
  12. 2025A Second-Order Perspective on Pruning at Initialization and Knowledge Transfercited by 0
4 more
  1. 2025Subtract the Corruption: Training-Data-Free Corrective Machine Unlearning using Task Arithmeticcited by 0
  2. 2026Learning to Recover Task Experts from a Multi-Task Merged Modelcited by 0
  3. 2026Fine-Tuning of Transformer models with Framescited by 0
  4. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0