mlsanityData to 7 Oct 2026

Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs

arXiv 2406.16797 ↗First posted 24 Jun 2024cs.CL cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Existing methods for adapting large language models (LLMs) to new tasks are not suited to multi-task adaptation because they modify all the model weights -- causing destructive interference between tasks. The resulting effects, such as catastrophic forgetting of earlier tasks, make it challenging to obtain good performance on multiple tasks at the same time. To mitigate this, we propose Lottery Ticket Adaptation (LoTA), a sparse adaptation method that identifies and optimizes only a sparse subnetwork of the model. We evaluate LoTA on a wide range of challenging tasks such as instruction following, reasoning, math, and summarization. LoTA obtains better performance than full fine-tuning and low-rank adaptation (LoRA), and maintains good performance even after training on other tasks -- thus, avoiding catastrophic forgetting. By extracting and fine-tuning over lottery tickets (or sparse task vectors), LoTA also enables model merging over highly dissimilar tasks. Our code is made publicly available at https://github.com/kiddyboots216/lottery-ticket-adaptation.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 74 of its 95 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

74 of 95 references.

References · 74

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  13. 2022Editing Models with Task Arithmeticcited by 1,076
  14. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  15. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  16. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  17. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  18. 2017A Survey on Multi-Task Learningcited by 838
  19. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  20. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  21. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  22. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  23. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  24. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  25. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  26. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  27. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  28. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  29. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  30. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  31. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  32. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  33. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  34. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  35. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  36. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  37. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  38. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 20; 2026: 4.

31 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  3. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
  4. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  5. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  6. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptationcited by 18
  7. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  8. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  9. 2025Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learningcited by 15
  10. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  11. 2025Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruningcited by 12
  12. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 10
19 more
  1. 2024S2FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsitycited by 10
  2. 2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
  3. 2025On Task Vectors and Gradientscited by 7
  4. 2025Context-Free Synthetic Data Mitigates Forgettingcited by 5
  5. 2025Alignment-Constrained Dynamic Pruning for LLMs: Identifying and Preserving Alignment-Critical Circuitscited by 4
  6. 2025Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updatescited by 4
  7. 2025TASO: Task-Aligned Sparse Optimization for Parameter-Efficient Model Adaptationcited by 3
  8. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Modelscited by 2
  9. 2025Model Merging Improves Zero-Shot Generalization in Bioacoustic Foundation Modelscited by 2
  10. 2026Towards Building Non-Fine-Tunable Foundation Modelscited by 2
  11. 2026The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVRcited by 2
  12. 2026PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspectivecited by 2
  13. 2025An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruningcited by 1
  14. 2025DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfercited by 1
  15. 2025Null-LoRA: Low-Rank Adaptation on Null Spacecited by 1
  16. 2026HELLoRA: Hot Experts Layer-Level Low-Rank Adaptation for Mixture-of-Experts Modelscited by 1
  17. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0
  18. 2025Model Connectomes: A Generational Approach to Data-Efficient Language Modelscited by 0
  19. 2025Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Expertscited by 0