mlsanityData to 7 Oct 2026

More Than Catastrophic Forgetting: Integrating General Capabilities For Domain-Specific LLMs

arXiv 2405.17830 ↗First posted 28 May 2024cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

The performance on general tasks decreases after Large Language Models (LLMs) are fine-tuned on domain-specific tasks, the phenomenon is known as Catastrophic Forgetting (CF). However, this paper presents a further challenge for real application of domain-specific LLMs beyond CF, called General Capabilities Integration (GCI), which necessitates the integration of both the general capabilities and domain knowledge within a single instance. The objective of GCI is not merely to retain previously acquired general capabilities alongside new domain knowledge, but to harmonize and utilize both sets of skills in a cohesive manner to enhance performance on domain-specific tasks. Taking legal domain as an example, we carefully design three groups of training and testing tasks without lacking practicability, and construct the corresponding datasets. To better incorporate general capabilities across domain-specific scenarios, we introduce ALoRA, which utilizes a multi-head attention module upon LoRA, facilitating direct information transfer from preceding tokens to the current one. This enhancement permits the representation to dynamically switch between domain-specific knowledge and general competencies according to the attention. Extensive experiments are conducted on the proposed tasks. The results exhibit the significance of our setting, and the effectiveness of our method.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

20 of 30 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2021Robust fine-tuning of zero-shot modelscited by 785
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  11. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  12. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
8 more
  1. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  2. 2023Continual Pre-training of Language Modelscited by 194
  3. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
  4. 2021LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5cited by 103
  5. 2022Continual Pre-Training Mitigates Forgetting in Language and Visioncited by 82
  6. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 48
  7. 2021Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mappingcited by 47
  8. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 9.

21 papers.

Sort:

By area:

  1. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 23
  2. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  3. 2024Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMscited by 11
  4. 2025Emotion-Qwen: A Unified Framework for Emotion and Vision Understandingcited by 10
  5. 2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
  6. 2025Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkishcited by 7
  7. 2025Challenges for Generative AI in Legal Reasoningcited by 7
  8. 2025ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuningcited by 4
  9. 2026Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesiscited by 4
  10. 2025Understanding the Dilemma of Unlearning for Large Language Modelscited by 3
  11. 2025Parameter Importance-Driven Continual Learning for Foundation Modelscited by 2
  12. 2026MixSD: Mixed Contextual Self-Distillation for Knowledge Injectioncited by 2
9 more
  1. 2026MemSFT: Mitigating Alignment Tax with an External Parametric Memorycited by 2
  2. 2024Enhancing Event Reasoning in Large Language Models through Instruction Fine-Tuning with Semantic Causal Graphscited by 1
  3. 2026"The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Frameworkcited by 1
  4. 2026Mol-Debate: Multi-Agent Debate Improves Structural Reasoning in Molecular Designcited by 1
  5. 2026Self-Specialized Teachers for Domain Post-Trainingcited by 1
  6. 2026CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptationcited by 1
  7. 2025ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretrainingcited by 0
  8. 2026Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillationcited by 0
  9. 2026ChainLoRA: Geometry-Preserving Task Vector Merging for Continual Learning in LLMscited by 0