# Dial-insight: Fine-tuning Large Language Models with High-Quality Domain-Specific Data Preventing Capability Collapse

arXiv: https://arxiv.org/abs/2403.09167 · HTML: https://mlsanity.com/paper/2403.09167/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 22 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization](https://mlsanity.com/paper/2108.13139/) (2021)
- [ChatHome: Development and Evaluation of a Domain-Specific Language Model for Home Renovation](https://mlsanity.com/paper/2307.15290/) (2023)
- [WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset](https://mlsanity.com/paper/2402.19282/) (2024)
- [The Transformative Influence of LLMs on Software Development & Developer Productivity](https://mlsanity.com/paper/2311.16429/) (2023)
- [Shai: A large language model for asset management](https://mlsanity.com/paper/2312.14203/) (2023)
- [Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing](https://mlsanity.com/paper/2311.14766/) (2023)

## Built on by (most cited first, 4 of 4)

- [Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report](https://mlsanity.com/paper/2504.21039/) (2025)
- [MoLoRAG: Bootstrapping Document Understanding via Multi-modal Logic-aware Retrieval](https://mlsanity.com/paper/2509.07666/) (2025)
- [Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2410.23111/) (2024)
- [A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent](https://mlsanity.com/paper/2509.19209/) (2025)

## Abstract (arXiv)

The efficacy of large language models (LLMs) is heavily dependent on the quality of the underlying data, particularly within specialized domains. A common challenge when fine-tuning LLMs for domain-specific applications is the potential degradation of the model's generalization capabilities. To address these issues, we propose a two-stage approach for the construction of production prompts designed to yield high-quality data. This method involves the generation of a diverse array of prompts that encompass a broad spectrum of tasks and exhibit a rich variety of expressions. Furthermore, we introduce a cost-effective, multi-dimensional quality assessment framework to ensure the integrity of the generated labeling data. Utilizing a dataset comprised of service provider and customer interactions from the real estate sector, we demonstrate a positive correlation between data quality and model performance. Notably, our findings indicate that the domain-specific proficiency of general LLMs can be enhanced through fine-tuning with data produced via our proposed method, without compromising their overall generalization abilities, even when exclusively domain-specific data is employed for fine-tuning.