# Legal Documents Drafting with Fine-Tuned Pre-Trained Large Language Model

arXiv: https://arxiv.org/abs/2406.04202 · HTML: https://mlsanity.com/paper/2406.04202/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 9 of 9)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Results](https://mlsanity.com/paper/2303.04715/) (2023)
- [Unsupervised Law Article Mining based on Deep Pre-Trained Language Representation Models with Application to the Italian Civil Code](https://mlsanity.com/paper/2112.03033/) (2021)

## Built on by (most cited first, 5 of 5)

- [HalluEntity: Benchmarking and Understanding Entity-Level Hallucination Detection](https://mlsanity.com/paper/2502.11948/) (2025)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [LicenseGPT: A Fine-tuned Foundation Model for Publicly Available Dataset License Compliance](https://mlsanity.com/paper/2501.00106/) (2024)
- [A Large Language Model-based Framework for Semi-Structured Tender Document Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.09077/) (2024)
- [Fine-tuning of Large Language Models for Domain-Specific Cybersecurity Knowledge](https://mlsanity.com/paper/2509.25241/) (2025)

## Abstract (arXiv)

With the development of large-scale Language Models (LLM), fine-tuning pre-trained LLM has become a mainstream paradigm for solving downstream tasks of natural language processing. However, training a language model in the legal field requires a large number of legal documents so that the language model can learn legal terminology and the particularity of the format of legal documents. The typical NLP approaches usually rely on many manually annotated data sets for training. However, in the legal field application, it is difficult to obtain a large number of manually annotated data sets, which restricts the typical method applied to the task of drafting legal documents. The experimental results of this paper show that not only can we leverage a large number of annotation-free legal documents without Chinese word segmentation to fine-tune a large-scale language model, but more importantly, it can fine-tune a pre-trained LLM on the local computer to achieve the generating legal document drafts task, and at the same time achieve the protection of information privacy and to improve information security issues.