mlsanityData to 14 Oct 2026

Confine to the Task, Anchor the Rest: Concept-Guided Retention Regularizer for Parameter-Efficient Fine-Tuning

arXiv 2610.17713 ↗First posted 13 Oct 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) has been widely used, yet the adapted model suffers catastrophic forgetting of pre-trained knowledge. While most existing work hardens the adapter by constraining where the weight update may lie, we note that constraining this weight channel does not constrain the activation channel. In this paper, we propose Concept-Guided Retention (CGR), a plug-and-play regularizer that constrains both for any PEFT base. CGR reads a frozen sparse autoencoder (SAE) and treats two disjoint concept sets differently. It softly confines the update to the task-concept subspace (via weight channel), and anchors a selected set of general-purpose concepts to the frozen model (via activation channel). We also provide anchor value bounds based on weight updates to understand how they interact, and show that both terms are necessary. On a CLIP vision panel, CGR significantly improves retention at comparable accuracy for every base, and the same pattern holds for world-knowledge retention under Llama-3.1-8B instruction tuning. Because the protected directions are named SAE concepts, CGR remains inspectable and controllable.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

24 of 39 references.

References · 24

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  3. 2016Learning without Forgettingcited by 2,496
  4. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  5. 2020Concept Bottleneck Modelscited by 1,032
  6. 2021Robust fine-tuning of zero-shot modelscited by 797
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  8. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  9. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  10. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  11. 2024LoRA Learns Less and Forgets Lesscited by 328
  12. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
12 more
  1. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 146
  2. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 109
  3. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  4. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 92
  5. 2025Model Unlearning via Sparse Autoencoder Subspace Guided Projectionscited by 18
  6. 2025Put the Space of LoRA Initialization to the Extreme to Preserve Pre-trained Knowledgecited by 16
  7. 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 14
  8. 2025Low-Rank Adapting Models for Sparse Autoencoderscited by 7
  9. 2025SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRAcited by 5
  10. 2025Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptationcited by 5
  11. 2026Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraintscited by 4
  12. 2026Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MERGETUNE: Continued Fine-Tuning of Vision-Language Models
  2. 2024Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves
  3. 2023Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models
  4. 2026Are Parameter-Efficient Fine-tuning Methods Really Different?
  5. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization
  6. 2026Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmetic
  7. 2024From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
  8. 2026Forget Less by Learning Together through Concept Consolidation