# Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models

arXiv: https://arxiv.org/abs/2403.02756 · HTML: https://mlsanity.com/paper/2403.02756/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 5 Mar 2024.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge](https://mlsanity.com/paper/2304.06975/) (2023)
- [CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2106.08087/) (2021)
- [XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters](https://mlsanity.com/paper/2305.12002/) (2023)
- [One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction](https://mlsanity.com/paper/2101.11427/) (2021)
- [Large Language Models are Diverse Role-Players for Summarization Evaluation](https://mlsanity.com/paper/2303.15078/) (2023)
- [BBT-Fin: Comprehensive Construction of Chinese Financial Domain Pre-trained Language Model, Corpus and Benchmark](https://mlsanity.com/paper/2302.09432/) (2023)
- [Continual Prompt Tuning for Dialog State Tracking](https://mlsanity.com/paper/2203.06654/) (2022)
- [COSPLAY: Concept Set Guided Personalized Dialogue Generation Across Both Party Personas](https://mlsanity.com/paper/2205.00872/) (2022)
- [TPE: Towards Better Compositional Reasoning over Conceptual Tools with Multi-persona Collaboration](https://mlsanity.com/paper/2309.16090/) (2023)
- [Analyzing the Forgetting Problem in the Pretrain-Finetuning of Dialogue Response Models](https://mlsanity.com/paper/1910.07117/) (2019)
- [Decoupled Training: Return of Frustratingly Easy Multi-Domain Learning](https://mlsanity.com/paper/2309.10302/) (2023)

## Built on by (most cited first, 10 of 10)

- [Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment](https://mlsanity.com/paper/2407.10804/) (2024)
- [Re-Thinking Process Mining in the AI-Based Agents Era](https://mlsanity.com/paper/2408.07720/) (2024)
- [Mixture-of-Instructions: Aligning Large Language Models via Mixture Prompting](https://mlsanity.com/paper/2404.18410/) (2024)
- [On-Air Deep Learning Integrated Semantic Inference Models for Enhanced Earth Observation Satellite Networks](https://mlsanity.com/paper/2409.15246/) (2024)
- [When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs](https://mlsanity.com/paper/2605.29420/) (2026)
- [DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining](https://mlsanity.com/paper/2410.00260/) (2024)
- [J6: Jacobian-Driven Role Attribution for Multi-Objective Prompt Optimization in LLMs](https://mlsanity.com/paper/2508.12086/) (2025)
- [PortAgent: LLM-driven Vehicle Dispatching Agent for Port Terminals](https://mlsanity.com/paper/2512.14417/) (2025)
- [Implicit Framing in Obstetric Counseling Notes: A Grounded LLM Pipeline on a VBAC-Eligible Cohort](https://mlsanity.com/paper/2604.23059/) (2026)
- [EvoMO-SR: Multiobjective LLM-based Evolution of Symbolic Expressions with substructure guidance](https://mlsanity.com/paper/2609.36187/) (2026)

## Abstract (arXiv)

The growing interest in Large Language Models (LLMs) for specialized applications has revealed a significant challenge: when tailored to specific domains, LLMs tend to experience catastrophic forgetting, compromising their general capabilities and leading to a suboptimal user experience. Additionally, crafting a versatile model for multiple domains simultaneously often results in a decline in overall performance due to confusion between domains. In response to these issues, we present the RolE Prompting Guided Multi-Domain Adaptation (REGA) strategy. This novel approach effectively manages multi-domain LLM adaptation through three key components: 1) Self-Distillation constructs and replays general-domain exemplars to alleviate catastrophic forgetting. 2) Role Prompting assigns a central prompt to the general domain and a unique role prompt to each specific domain to minimize inter-domain confusion during training. 3) Role Integration reuses and integrates a small portion of domain-specific data to the general-domain data, which are trained under the guidance of the central prompt. The central prompt is used for a streamlined inference process, removing the necessity to switch prompts for different domains. Empirical results demonstrate that REGA effectively alleviates catastrophic forgetting and inter-domain confusion. This leads to improved domain-specific performance compared to standard fine-tuned models, while still preserving robust general capabilities.