mlsanityData to 7 Oct 2026

An overview of domain-specific foundation model: key technologies, applications and challenges

arXiv 2409.04267 ↗First posted 6 Sep 2024cs.AI cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

The impressive performance of ChatGPT and other foundation-model-based products in human language understanding has prompted both academia and industry to explore how these models can be tailored for specific industries and application scenarios. This process, known as the customization of domain-specific foundation models (FMs), addresses the limitations of general-purpose models, which may not fully capture the unique patterns and requirements of domain-specific data. Despite its importance, there is a notable lack of comprehensive overview papers on building domain-specific FMs, while numerous resources exist for general-purpose models. To bridge this gap, this article provides a timely and thorough overview of the methodology for customizing domain-specific FMs. It introduces basic concepts, outlines the general architecture, and surveys key methods for constructing domain-specific models. Furthermore, the article discusses various domains that can benefit from these specialized models and highlights the challenges ahead. Through this overview, we aim to offer valuable guidance and reference for researchers and practitioners from diverse fields to develop their own customized FMs.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 157 of its 208 references that are in the map. First posted on arXiv 6 Sep 2024.

Builds on

157 of 208 references.

References · 157

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2024The Llama 3 Herd of Modelscited by 14,192
  10. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  11. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  12. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
38 more
  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  13. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  14. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  15. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  16. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  17. 2020Longformer: The Long-Document Transformercited by 3,362
  18. 2023Qwen Technical Reportcited by 3,273
  19. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  20. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  21. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  22. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  23. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  24. 2023A Survey of Large Language Modelscited by 2,537
  25. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  26. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  27. 2022Emergent Abilities of Large Language Modelscited by 2,483
  28. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  29. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  30. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  31. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  32. 2024Mixtral of Expertscited by 1,785
  33. 2019Towards VQA Models That Can Readcited by 1,696
  34. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  35. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  36. 2020Reformer: The Efficient Transformercited by 1,658
  37. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  38. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411

50 most cited of 157 All 157 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 5.

19 papers.

Sort:

By area:

  1. 2024CSI-BERT2: A BERT-inspired Framework for Efficient CSI Prediction and Classification in Wireless Communication and Sensingcited by 11
  2. 2025Surgical Scene Understanding in the Era of Foundation AI Models: A Comprehensive Reviewcited by 8
  3. 2024KNN-MMD: Cross Domain Wireless Sensing via Local Distribution Alignmentcited by 3
  4. 2025GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoningcited by 3
  5. 2025Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidancecited by 3
  6. 2024Data-Efficient CLIP-Powered Dual-Branch Networks for Source-Free Unsupervised Domain Adaptationcited by 2
  7. 2025Breaking Down Bias: On The Limits of Generalizable Pruning Strategiescited by 2
  8. 2025Confident RAG: Enhancing the Performance of LLMs for Mathematics Question Answering through Multi-Embedding and Confidence Scoringcited by 2
  9. 2025STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Predictioncited by 1
  10. 2025DK-Root: A Joint Data-and-Knowledge-Driven Framework for Root Cause Analysis of QoE Degradations in Mobile Networkscited by 1
  11. 2026Odyssey: Constructing Verifiable Local Truth-Preserving Foundation Modelscited by 1
  12. 2026Error-Aware Reverse Auction Mechanism for Large Language Model Routingcited by 1
7 more
  1. 2025Label Unbalance in High-frequency Tradingcited by 0
  2. 2025PB-IAD: Utilizing multimodal foundation models for semantic industrial anomaly detection in dynamic manufacturing environmentscited by 0
  3. 2025Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approachcited by 0
  4. 2025AutoFed: Personalized Federated Traffic Prediction via Adaptive Promptcited by 0
  5. 2026Multi-Modal LLM based Image Captioning in ICT: Bridging the Gap Between General and Industry Domaincited by 0
  6. 2026The Economics of AI Supply Chain Regulationcited by 0
  7. 2026ERSkill: Evolving for Skill-Guided Adaptive Memory Retrievalcited by 0