mlsanityData to 7 Oct 2026

Ability Transfer and Recovery via Modularized Parameters Localization

arXiv 2601.09398 ↗First posted 14 Jan 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models can be continually pre-trained or fine-tuned to improve performance in specific domains, languages, or skills, but this specialization often degrades other capabilities and may cause catastrophic forgetting. We investigate how abilities are distributed within LLM parameters by analyzing module activations under domain- and language-specific inputs for closely related models. Across layers and modules, we find that ability-related activations are highly concentrated in a small set of channels (typically <5%), and these channels are largely disentangled with good sufficiency and stability. Building on these observations, we propose ACT (Activation-Guided Channel-wise Ability Transfer), which localizes ability-relevant channels via activation differences and selectively transfers only the corresponding parameters, followed by lightweight fine-tuning for compatibility. Experiments on multilingual mathematical and scientific reasoning show that ACT can recover forgotten abilities while preserving retained skills. It can also merge multiple specialized models to integrate several abilities into a single model with minimal interference. Our code and data will be publicly released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 14 Jan 2026.

Builds on

25 of 37 references.

References · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
13 more
  1. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  2. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  3. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  4. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  5. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  6. 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 74
  7. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 57
  8. 2025MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoningcited by 56
  9. 2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
  10. 2019A Unifying Bayesian View of Continual Learningcited by 38
  11. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  12. 2024Encourage or Inhibit Monosemanticity? Revisit Monosemanticity from a Feature Decorrelation Perspectivecited by 10
  13. 2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
  2. 2026Post-Training Leaves Behavioral Shadows on Unrelated Decisions
  3. 2025Emergent Abilities in Large Language Models: A Survey
  4. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  5. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inference
  6. 2025Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Training
  7. 2026Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning
  8. 2025The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs