mlsanityData to 14 Oct 2026

Learn Here, Move Less Elsewhere: Input-Conditioned Plasticity from Retained-Domain Activation Atlases

arXiv 2609.34478 ↗First posted 28 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Task-specific fine-tuning can rewrite a language model's answers beyond the training task, complicating updates that must preserve existing behavior. We introduce ATLAS, which turns retained-domain representations into an input-dependent rule for task adaptation. An activation atlas supplies local reference centers and directional filters to a shared low-rank residual. Target supervision learns the residual, while retained geometry shapes its action throughout training and inference. On Qwen3-8B, ATLAS achieves lower mean retained-output Kullback-Leibler (KL) divergence than all seven published baselines at shared coding-performance requirements, with consistent advantages across multiple training seeds. Structural comparisons identify the contributions of retained reference states and directional conditioning, and answer-level analyses show fewer rewritten mathematical answers and more stable commonsense choices. Experiments spanning five backbones and two retained domains further demonstrate coding gains with reduced retained-output movement. With compact storage and modest decoding overhead, ATLAS provides a practical mechanism for acquiring specialized skills while maintaining continuity in existing responses.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 29 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

26 of 29 references.

References · 26

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2016Learning without Forgettingcited by 2,496
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  12. 2018Efficient Lifelong Learning with A-GEMcited by 1,058
14 more
  1. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  2. 2021Gradient Projection Memory for Continual Learningcited by 348
  3. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  4. 2020Positive-Congruent Training: Towards Regression-Free Model Updatescited by 57
  5. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 24
  6. 2025OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuningcited by 21
  7. 2024SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixturecited by 19
  8. 2025Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledgecited by 16
  9. 2024MUSCLE: A Model Update Strategy for Compatible LLM Evolutioncited by 15
  10. 2025Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learningcited by 14
  11. 2024Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Modelscited by 9
  12. 2023Backward Compatibility During Data Updates by Weight Interpolationcited by 7
  13. 2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 6
  14. 2026PASs-MoE: Mitigating Misaligned Co-drift among Router and Experts via Pathway Activation Subspaces for Continual Learningcited by 4

Built on by

Not cited yet.