# Reformulating Domain Adaptation of Large Language Models as Adapt-Retrieve-Revise: A Case Study on Chinese Legal Domain

arXiv: https://arxiv.org/abs/2310.03328 · HTML: https://mlsanity.com/paper/2310.03328/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services](https://mlsanity.com/paper/2309.11325/) (2023)
- [JEC-QA: A Legal-Domain Question Answering Dataset](https://mlsanity.com/paper/1911.12011/) (2019)

## Built on by (most cited first, 12 of 12)

- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [Recent Advances of Foundation Language Models-based Continual Learning: A Survey](https://mlsanity.com/paper/2405.18653/) (2024)
- [Large Language Models in Cybersecurity: State-of-the-Art](https://mlsanity.com/paper/2402.00891/) (2024)
- [Do LLMs Really Adapt to Domains? An Ontology Learning Perspective](https://mlsanity.com/paper/2407.19998/) (2024)
- [When Continue Learning Meets Multimodal Large Language Model: A Survey](https://mlsanity.com/paper/2503.01887/) (2025)
- [Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey](https://mlsanity.com/paper/2404.00990/) (2024)
- [Beyond Guilt: Legal Judgment Prediction with Trichotomous Reasoning](https://mlsanity.com/paper/2412.14588/) (2024)
- [Automating Governing Knowledge Commons and Contextual Integrity (GKC-CI) Privacy Policy Annotations with Large Language Models](https://mlsanity.com/paper/2311.02192/) (2023)
- [DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining](https://mlsanity.com/paper/2410.00260/) (2024)
- [VLLaVO: Mitigating Visual Gap through LLMs](https://mlsanity.com/paper/2401.03253/) (2024)
- [From Random to Informed Data Selection: A Diversity-Based Approach to Optimize Human Annotation and Few-Shot Learning](https://mlsanity.com/paper/2401.13229/) (2024)
- [Knowledge Acquisition on Mass-shooting Events via LLMs for AI-Driven Justice](https://mlsanity.com/paper/2504.12545/) (2025)

## Abstract (arXiv)

While large language models (LLMs) like GPT-4 have recently demonstrated astonishing zero-shot capabilities in general domain tasks, they often generate content with hallucinations in specific domains such as Chinese law, hindering their application in these areas. This is typically due to the absence of training data that encompasses such a specific domain, preventing GPT-4 from acquiring in-domain knowledge. A pressing challenge is that it's not plausible to continue training LLMs of such scale on in-domain data. This paper introduces a simple and effective domain adaptation framework for GPT-4 by reformulating generation as an adapt-retrieve-revise process. The initial step is to adapt an affordable 7B LLM to the target domain by continuing learning on in-domain data. When solving a task, we leverage the adapted LLM to generate a draft answer given a task query. Then, the draft answer will be used to retrieve supporting evidence candidates from an external in-domain knowledge base. Finally, the draft answer and retrieved evidence are concatenated into a whole prompt to let GPT-4 assess the evidence and revise the draft answer to generate the final answer. Our proposal combines the advantages of the efficiency of adapting a smaller 7B model with the evidence-assessing capability of GPT-4 and effectively prevents GPT-4 from generating hallucinatory content. In the zero-shot setting of four Chinese legal tasks, our method improves accuracy by 33.3% compared to the direct generation by GPT-4. When compared to two stronger retrieval-based baselines, our method outperforms them by 15.4% and 23.9%. Our code will be released