# Fine-tuning and Utilization Methods of Domain-specific LLMs

arXiv: https://arxiv.org/abs/2401.02981 · HTML: https://mlsanity.com/paper/2401.02981/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 34 references that are in the map.
- First posted on arXiv 1 Jan 2024.

## Builds on (most cited first, 7 of 7)

- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [A Study on the Implementation of Generative AI Services Using an Enterprise Data-Based LLM Application Architecture](https://mlsanity.com/paper/2309.01105/) (2023)

## Built on by (most cited first, 30 of 36)

- [Revolutionizing Finance with LLMs: An Overview of Applications and Insights](https://mlsanity.com/paper/2401.11641/) (2024)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios](https://mlsanity.com/paper/2412.08972/) (2024)
- [Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety](https://mlsanity.com/paper/2505.06843/) (2025)
- [Only a Little to the Left: A Theory-grounded Measure of Political Bias in Large Language Models](https://mlsanity.com/paper/2503.16148/) (2025)
- [Do Language Models Understand Time?](https://mlsanity.com/paper/2412.13845/) (2024)
- [Enhancing Supply Chain Visibility with Knowledge Graphs and Large Language Models](https://mlsanity.com/paper/2408.07705/) (2024)
- [GeoCode-GPT: A Large Language Model for Geospatial Code Generation Tasks](https://mlsanity.com/paper/2410.17031/) (2024)
- [Domaino1s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains](https://mlsanity.com/paper/2501.14431/) (2025)
- [A Lightweight Large Language Model-Based Multi-Agent System for 2D Frame Structural Analysis](https://mlsanity.com/paper/2510.05414/) (2025)
- [Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models](https://mlsanity.com/paper/2408.05241/) (2024)
- [Education in the Era of Neurosymbolic AI](https://mlsanity.com/paper/2411.12763/) (2024)
- [All-in-One Tuning and Structural Pruning for Domain-Specific LLMs](https://mlsanity.com/paper/2412.14426/) (2024)
- [MARK: Memory Augmented Refinement of Knowledge](https://mlsanity.com/paper/2505.05177/) (2025)
- [Baba is LLM: Reasoning in a Game with Dynamic Rules](https://mlsanity.com/paper/2506.19095/) (2025)
- [EDCO: Dynamic Curriculum Orchestration for Domain-specific Large Language Model Fine-tuning](https://mlsanity.com/paper/2601.03725/) (2026)
- [ALKAFI-LLAMA3: Fine-Tuning LLMs for Precise Legal Understanding in Palestine](https://mlsanity.com/paper/2412.14771/) (2024)
- [Experiences with Content Development and Assessment Design in the Era of GenAI](https://mlsanity.com/paper/2503.00081/) (2025)
- [Fine-Tuning LLMs for Report Summarization: Analysis on Supervised and Unsupervised Data](https://mlsanity.com/paper/2503.10676/) (2025)
- [Predicting Human Choice Between Textually Described Lotteries](https://mlsanity.com/paper/2503.14004/) (2025)
- [Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation](https://mlsanity.com/paper/2506.15702/) (2025)
- [Achieving Trustworthy Real-Time Decision Support Systems with Low-Latency Interpretable AI Models](https://mlsanity.com/paper/2506.20018/) (2025)
- [A Survey: Towards Privacy and Security in Mobile Large Language Models](https://mlsanity.com/paper/2509.02411/) (2025)
- [ASVRI-Legal: Fine-Tuning LLMs with Retrieval Augmented Generation for Enhanced Legal Regulation](https://mlsanity.com/paper/2511.03563/) (2025)
- [Jal Anveshak: Prediction of fishing zones using fine-tuned LlaMa 2](https://mlsanity.com/paper/2411.10050/) (2024)
- [Efficient Learning Content Retrieval with Knowledge Injection](https://mlsanity.com/paper/2412.00125/) (2024)
- [MUST-RAG: MUSical Text Question Answering with Retrieval Augmented Generation](https://mlsanity.com/paper/2507.23334/) (2025)
- [Automated Item Neutralization for Non-Cognitive Scales: A Large Language Model Approach to Reducing Social-Desirability Bias](https://mlsanity.com/paper/2509.19314/) (2025)
- [Bayesian Network Fusion of Large Language Models for Sentiment Analysis](https://mlsanity.com/paper/2510.26484/) (2025)

## Abstract (arXiv)

Recent releases of pre-trained Large Language Models (LLMs) have gained considerable traction, yet research on fine-tuning and employing domain-specific LLMs remains scarce. This study investigates approaches for fine-tuning and leveraging domain-specific LLMs, highlighting trends in LLMs, foundational models, and methods for domain-specific pre-training. Focusing on the financial sector, it details dataset selection, preprocessing, model choice, and considerations crucial for LLM fine-tuning in finance. Addressing the unique characteristics of financial data, the study explores the construction of domain-specific vocabularies and considerations for security and regulatory compliance. In the practical application of LLM fine-tuning, the study outlines the procedure and implementation for generating domain-specific LLMs in finance. Various financial cases, including stock price prediction, sentiment analysis of financial news, automated document processing, research, information extraction, and customer service enhancement, are exemplified. The study explores the potential of LLMs in the financial domain, identifies limitations, and proposes directions for improvement, contributing valuable insights for future research. Ultimately, it advances natural language processing technology in business, suggesting proactive LLM utilization in financial services across industries.