# Fine Tuning LLM for Enterprise: Practical Guidelines and Recommendations

arXiv: https://arxiv.org/abs/2404.10779 · HTML: https://mlsanity.com/paper/2404.10779/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 23 Mar 2024.

## Builds on (most cited first, 7 of 7)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [A Study of BFLOAT16 for Deep Learning Training](https://mlsanity.com/paper/1905.12322/) (2019)
- [A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models](https://mlsanity.com/paper/2309.11674/) (2023)
- [A Study on the Implementation of Generative AI Services Using an Enterprise Data-Based LLM Application Architecture](https://mlsanity.com/paper/2309.01105/) (2023)

## Built on by (most cited first, 28 of 28)

- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness](https://mlsanity.com/paper/2410.21331/) (2024)
- [Metamorphic Testing of Large Language Models for Natural Language Processing](https://mlsanity.com/paper/2511.02108/) (2025)
- [Textualized and Feature-based Models for Compound Multimodal Emotion Recognition in the Wild](https://mlsanity.com/paper/2407.12927/) (2024)
- [Operating Room Workflow Analysis via Reasoning Segmentation over Digital Twins](https://mlsanity.com/paper/2503.21054/) (2025)
- [DataLab: A Unified Platform for LLM-Powered Business Intelligence](https://mlsanity.com/paper/2412.02205/) (2024)
- [Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models](https://mlsanity.com/paper/2408.05241/) (2024)
- [GoNoGo: An Efficient LLM-based Multi-Agent System for Streamlining Automotive Software Release Decision-Making](https://mlsanity.com/paper/2408.09785/) (2024)
- [Opportunities and Challenges of Generative-AI in Finance](https://mlsanity.com/paper/2410.15653/) (2024)
- [Rethinking Scale: The Efficacy of Fine-Tuned Open-Source LLMs in Large-Scale Reproducible Social Science Research](https://mlsanity.com/paper/2411.00890/) (2024)
- [GateLens: A Reasoning-Enhanced LLM Agent for Automotive Software Release Analytics](https://mlsanity.com/paper/2503.21735/) (2025)
- [Mamba State-Space Models Are Lyapunov-Stable Learners](https://mlsanity.com/paper/2406.00209/) (2024)
- [Combining Evidence and Reasoning for Biomedical Fact-Checking](https://mlsanity.com/paper/2509.13879/) (2025)
- [Use Me Wisely: AI-Driven Assessment for LLM Prompting Skills Development](https://mlsanity.com/paper/2503.02532/) (2025)
- [Deep literature reviews: an application of fine-tuned language models to migration research](https://mlsanity.com/paper/2504.13685/) (2025)
- [Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization System](https://mlsanity.com/paper/2505.01315/) (2025)
- [Decoding BACnet Packets: A Large Language Model Approach for Packet Interpretation](https://mlsanity.com/paper/2407.15428/) (2024)
- [GridCodex: A RAG-Driven AI Framework for Power Grid Code Reasoning and Compliance](https://mlsanity.com/paper/2508.12682/) (2025)
- [Efficient Toxicity Detection in Gaming Chats: A Comparative Study of Embeddings, Fine-Tuned Transformers and LLMs](https://mlsanity.com/paper/2510.17924/) (2025)
- [Creating a Gen-AI based Track and Trace Assistant MVP (SuperTracy) for PostNL](https://mlsanity.com/paper/2409.02711/) (2024)
- [Leveraging LLMs for Predictive Insights in Food Policy and Behavioral Interventions](https://mlsanity.com/paper/2411.08563/) (2024)
- [Efficient Learning Content Retrieval with Knowledge Injection](https://mlsanity.com/paper/2412.00125/) (2024)
- [Interweaving Memories of a Siamese Large Language Model](https://mlsanity.com/paper/2412.17383/) (2024)
- [Perturbing the Derivative: Wild Refitting for Model-Free Evaluation of Machine Learning Models under Bregman Losses](https://mlsanity.com/paper/2509.02476/) (2025)
- [A Human-Centred Architecture for Large Language Models-Cognitive Assistants in Manufacturing within Quality Management Systems](https://mlsanity.com/paper/2603.16325/) (2026)
- [New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Framework](https://mlsanity.com/paper/2604.09940/) (2026)
- [Gradient Transformer: Learning to Generate Updates for LLMs](https://mlsanity.com/paper/2605.27591/) (2026)
- [LCAi: Life Cycle Assessment with big data fusion and retrieval-augmented generation-assisted interpretation](https://mlsanity.com/paper/2606.26857/) (2026)

## Abstract (arXiv)

There is a compelling necessity from enterprises for fine tuning LLMs (Large Language Models) o get them trained on proprietary domain knowledge. The challenge is to imbibe the LLMs with domain specific knowledge using the most optimial resource and cost and in the best possible time. Many enterprises rely on RAG (Retrieval Augmented Generation) which does not need LLMs to be ine-tuned but they are limited by the quality of vector databases and their retrieval capabilities rather than the intrinsic capabilities of the LLMs themselves. In our current work we focus on fine tuning LLaMA, an open source LLM using proprietary documents and code from an enterprise repository and use the fine tuned models to evaluate the quality of responses. As part of this work, we aim to guide beginners on how to start with fine tuning an LLM for documentation and code by making educated guesses on size of GPU required and options that are available for formatting the data. We also propose pre processing recipes for both documentation and code to prepare dataset in different formats. The proposed methods of data preparation for document datasets are forming paragraph chunks, forming question and answer pairs and forming keyword and paragraph chunk pairs. For code dataset we propose forming summary and function pairs. Further, we qualitatively evaluate the results of the models for domain specific queries. Finally, we also propose practical guidelines and recommendations for fine tuning LLMs.