# Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey

arXiv: https://arxiv.org/abs/2305.18703 · HTML: https://mlsanity.com/paper/2305.18703/

## Key facts

- Cited by 135 papers in the map (data through 14 Oct 2026).
- Builds on 230 of its 363 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 30 of 230)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)

## Built on by (most cited first, 30 of 135)

- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [Language Models with Conformal Factuality Guarantees](https://mlsanity.com/paper/2402.10978/) (2024)
- [HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction](https://mlsanity.com/paper/2408.04948/) (2024)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks](https://mlsanity.com/paper/2405.15179/) (2024)
- [A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning](https://mlsanity.com/paper/2408.07057/) (2024)
- [SMART: Self-Aware Agent for Tool Overuse Mitigation](https://mlsanity.com/paper/2502.11435/) (2025)
- [Uncertainty Quantification for In-Context Learning of Large Language Models](https://mlsanity.com/paper/2402.10189/) (2024)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models](https://mlsanity.com/paper/2403.03432/) (2024)
- [CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solving](https://mlsanity.com/paper/2404.17729/) (2024)
- [BiomedRAG: A Retrieval Augmented Large Language Model for Biomedicine](https://mlsanity.com/paper/2405.00465/) (2024)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [Large Language Models for Spatial Trajectory Patterns Mining](https://mlsanity.com/paper/2310.04942/) (2023)
- [ChatHome: Development and Evaluation of a Domain-Specific Language Model for Home Renovation](https://mlsanity.com/paper/2307.15290/) (2023)
- [Efficient Continual Pre-training by Mitigating the Stability Gap](https://mlsanity.com/paper/2406.14833/) (2024)
- [LLMind: Orchestrating AI and IoT with LLM for Complex Task Execution](https://mlsanity.com/paper/2312.09007/) (2023)
- [Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Data](https://mlsanity.com/paper/2402.12869/) (2024)
- [Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts](https://mlsanity.com/paper/2406.12034/) (2024)
- [Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agenda](https://mlsanity.com/paper/2404.09554/) (2024)
- [Mamba in Vision: A Comprehensive Survey of Techniques and Applications](https://mlsanity.com/paper/2410.03105/) (2024)
- [Large Action Models: From Inception to Implementation](https://mlsanity.com/paper/2412.10047/) (2024)
- [PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generation](https://mlsanity.com/paper/2501.11551/) (2025)
- [FinMTEB: Finance Massive Text Embedding Benchmark](https://mlsanity.com/paper/2502.10990/) (2025)
- [MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models](https://mlsanity.com/paper/2405.13053/) (2024)
- [Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs](https://mlsanity.com/paper/2409.19759/) (2024)
- [Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models](https://mlsanity.com/paper/2502.11075/) (2025)
- [Learning Facts at Scale with Active Reading](https://mlsanity.com/paper/2508.09494/) (2025)
- [Do We Need Domain-Specific Embedding Models? An Empirical Investigation](https://mlsanity.com/paper/2409.18511/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have significantly advanced the field of natural language processing (NLP), providing a highly useful, task-agnostic foundation for a wide range of applications. However, directly applying LLMs to solve sophisticated problems in specific domains meets many hurdles, caused by the heterogeneity of domain data, the sophistication of domain knowledge, the uniqueness of domain objectives, and the diversity of the constraints (e.g., various social norms, cultural conformity, religious beliefs, and ethical standards in the domain applications). Domain specification techniques are key to make large language models disruptive in many applications. Specifically, to solve these hurdles, there has been a notable increase in research and practices conducted in recent years on the domain specialization of LLMs. This emerging field of study, with its substantial potential for impact, necessitates a comprehensive and systematic review to better summarize and guide ongoing work in this area. In this article, we present a comprehensive survey on domain specification techniques for large language models, an emerging direction critical for large language model applications. First, we propose a systematic taxonomy that categorizes the LLM domain-specialization techniques based on the accessibility to LLMs and summarizes the framework for all the subcategories as well as their relations and differences to each other. Second, we present an extensive taxonomy of critical application domains that can benefit dramatically from specialized LLMs, discussing their practical significance and open challenges. Last, we offer our insights into the current research status and future trends in this area.