mlsanityData to 14 Oct 2026

Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey

arXiv 2305.18703 ↗First posted 30 May 2023cs.CL cs.AIcited by 135

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have significantly advanced the field of natural language processing (NLP), providing a highly useful, task-agnostic foundation for a wide range of applications. However, directly applying LLMs to solve sophisticated problems in specific domains meets many hurdles, caused by the heterogeneity of domain data, the sophistication of domain knowledge, the uniqueness of domain objectives, and the diversity of the constraints (e.g., various social norms, cultural conformity, religious beliefs, and ethical standards in the domain applications). Domain specification techniques are key to make large language models disruptive in many applications. Specifically, to solve these hurdles, there has been a notable increase in research and practices conducted in recent years on the domain specialization of LLMs. This emerging field of study, with its substantial potential for impact, necessitates a comprehensive and systematic review to better summarize and guide ongoing work in this area. In this article, we present a comprehensive survey on domain specification techniques for large language models, an emerging direction critical for large language model applications. First, we propose a systematic taxonomy that categorizes the LLM domain-specialization techniques based on the accessibility to LLMs and summarizes the framework for all the subcategories as well as their relations and differences to each other. Second, we present an extensive taxonomy of critical application domains that can benefit dramatically from specialized LLMs, discussing their practical significance and open challenges. Last, we offer our insights into the current research status and future trends in this area.

Key facts. Cited by 135 papers in the map (data through 14 Oct 2026). Builds on 230 of its 363 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

230 of 363 references.

References · 230

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
  13. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  14. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  15. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  16. 2023A Survey of Large Language Modelscited by 2,576
  17. 2022Emergent Abilities of Large Language Modelscited by 2,512
  18. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  19. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  20. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  21. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  22. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  23. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  24. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  25. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  26. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  27. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  28. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  29. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  30. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  31. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  32. 2017Continual Learning with Deep Generative Replaycited by 1,158
  33. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  34. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  35. 2022Mass-Editing Memory in a Transformercited by 1,001
  36. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  37. 2023BloombergGPT: A Large Language Model for Financecited by 939
  38. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938

50 most cited of 230 All 230 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 54; 2025: 39; 2026: 30.

135 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  3. 2024Language Models with Conformal Factuality Guaranteescited by 131
  4. 2024HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extractioncited by 93
  5. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
  6. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  7. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  8. 2025SMART: Self-Aware Agent for Tool Overuse Mitigationcited by 52
  9. 2024Uncertainty Quantification for In-Context Learning of Large Language Modelscited by 45
  10. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  11. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
  12. 2024CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solvingcited by 31
38 more
  1. 2024BiomedRAG: A Retrieval Augmented Large Language Model for Biomedicinecited by 31
  2. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  3. 2023Large Language Models for Spatial Trajectory Patterns Miningcited by 27
  4. 2023ChatHome: Development and Evaluation of a Domain-Specific Language Model for Home Renovationcited by 25
  5. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  6. 2023LLMind: Orchestrating AI and IoT with LLM for Complex Task Executioncited by 24
  7. 2024Exploring the Impact of Table-to-Text Methods on Augmenting LLM-based Question Answering with Domain Hybrid Datacited by 24
  8. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 23
  9. 2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
  10. 2024Mamba in Vision: A Comprehensive Survey of Techniques and Applicationscited by 18
  11. 2024Large Action Models: From Inception to Implementationcited by 18
  12. 2025PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generationcited by 18
  13. 2025FinMTEB: Finance Massive Text Embedding Benchmarkcited by 18
  14. 2024MeteoRA: Multiple-tasks Embedded LoRA for Large Language Modelscited by 16
  15. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMscited by 15
  16. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  17. 2025Learning Facts at Scale with Active Readingcited by 13
  18. 2024Do We Need Domain-Specific Embedding Models? An Empirical Investigationcited by 12
  19. 2025TSPulse: Tiny Pre-Trained Models with Disentangled Representations for Rapid Time-Series Analysiscited by 12
  20. 2025When LLM Meets Time Series: Can LLMs Perform Multi-Step Time Series Reasoning and Inferencecited by 12
  21. 2024Large Language Models for Social Networks: Applications, Challenges, and Solutionscited by 11
  22. 2024LASSI: An LLM-based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codescited by 10
  23. 2024TEG-DB: A Comprehensive Dataset and Benchmark of Textual-Edge Graphscited by 9
  24. 2023LLaMA-E: Empowering E-commerce Authoring with Object-Interleaved Instruction Followingcited by 8
  25. 2023Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertaintycited by 8
  26. 2024TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Modelscited by 8
  27. 2024ItD: Large Language Models Can Teach Themselves Induction through Deductioncited by 8
  28. 2024Only-IF:Revealing the Decisive Effect of Instruction Diversity on Generalizationcited by 8
  29. 2025Toward Structured Knowledge Reasoning: Contrastive Retrieval-Augmented Generation on Experiencecited by 8
  30. 2025Large Language Models and Their Applications in Roadway Safety and Mobility Enhancement: A Comprehensive Reviewcited by 8
  31. 2025Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunkingcited by 8
  32. 2023A Reliable Knowledge Processing Framework for Combustion Science using Foundation Modelscited by 7
  33. 2024A Comparative Study of DSPy Teleprompter Algorithms for Aligning Large Language Models Evaluation Metrics to Human Evaluationcited by 7
  34. 2025Challenges for Generative AI in Legal Reasoningcited by 7
  35. 2023POND: Multi-Source Time Series Domain Adaptation with Information-Aware Prompt Tuningcited by 6
  36. 2024Multi-modal Causal Structure Learning and Root Cause Analysiscited by 6
  37. 2024LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Languagecited by 6
  38. 2024LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Contentcited by 6

50 most cited of 135 All 135 (JSON)