# Lawyer LLaMA Technical Report

arXiv: https://arxiv.org/abs/2305.15062 · HTML: https://mlsanity.com/paper/2305.15062/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 27 of 27)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Episodic Memory in Lifelong Language Learning](https://mlsanity.com/paper/1906.01076/) (2019)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting](https://mlsanity.com/paper/2004.12651/) (2020)
- [DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task](https://mlsanity.com/paper/2304.01097/) (2023)
- [JEC-QA: A Legal-Domain Question Answering Dataset](https://mlsanity.com/paper/1911.12011/) (2019)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension](https://mlsanity.com/paper/1904.09679/) (2019)
- [Distinguish Confusing Law Articles for Legal Judgment Prediction](https://mlsanity.com/paper/2004.02557/) (2020)
- [Learning to Predict Charges for Criminal Cases with Legal Basis](https://mlsanity.com/paper/1707.09168/) (2017)
- [CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model](https://mlsanity.com/paper/2003.01355/) (2020)
- [Can BERT Refrain from Forgetting on Sequential Tasks? A Probing Study](https://mlsanity.com/paper/2303.01081/) (2023)

## Built on by (most cited first, 30 of 54)

- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://mlsanity.com/paper/2312.05934/) (2023)
- [DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services](https://mlsanity.com/paper/2309.11325/) (2023)
- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [Adapting Large Language Models to Domains via Reading Comprehension](https://mlsanity.com/paper/2309.09530/) (2023)
- [EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commerce](https://mlsanity.com/paper/2308.06966/) (2023)
- [LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation](https://mlsanity.com/paper/2403.01131/) (2024)
- [A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection](https://mlsanity.com/paper/2310.06498/) (2023)
- [LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Model](https://mlsanity.com/paper/2406.04614/) (2024)
- [Integration of Large Language Models and Federated Learning](https://mlsanity.com/paper/2307.08925/) (2023)
- [SciLitLLM: How to Adapt LLMs for Scientific Literature Understanding](https://mlsanity.com/paper/2408.15545/) (2024)
- [MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model](https://mlsanity.com/paper/2409.07486/) (2024)
- [PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation](https://mlsanity.com/paper/2407.05721/) (2024)
- [Efficient Continual Pre-training by Mitigating the Stability Gap](https://mlsanity.com/paper/2406.14833/) (2024)
- [PLLaMa: An Open-source Large Language Model for Plant Science](https://mlsanity.com/paper/2401.01600/) (2024)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensation](https://mlsanity.com/paper/2312.17276/) (2023)
- [Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Finance](https://mlsanity.com/paper/2410.01109/) (2024)
- [LawLuo: A Multi-Agent Collaborative Framework for Multi-Round Chinese Legal Consultation](https://mlsanity.com/paper/2407.16252/) (2024)
- [MIND: Math Informed syNthetic Dialogues for Pretraining LLMs](https://mlsanity.com/paper/2410.12881/) (2024)
- [Domaino1s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains](https://mlsanity.com/paper/2501.14431/) (2025)
- [Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Design](https://mlsanity.com/paper/2508.17573/) (2025)
- [A Survey on Symbolic Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2408.10210/) (2024)
- [LawGPT: Knowledge-Guided Data Generation and Its Application to Legal LLM](https://mlsanity.com/paper/2502.06572/) (2025)
- [HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory System](https://mlsanity.com/paper/2603.14807/) (2026)
- [A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences](https://mlsanity.com/paper/2503.00841/) (2025)
- [Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey](https://mlsanity.com/paper/2404.00990/) (2024)
- [FedJudge: Federated Legal Large Language Model](https://mlsanity.com/paper/2309.08173/) (2023)
- [InternLM-Law: An Open Source Chinese Legal Large Language Model](https://mlsanity.com/paper/2406.14887/) (2024)

## Abstract (arXiv)

Large Language Models (LLMs), like LLaMA, have exhibited remarkable performance across various tasks. Nevertheless, when deployed to specific domains such as law or medicine, the models still confront the challenge of a deficiency in domain-specific knowledge and an inadequate capability to leverage that knowledge to resolve domain-related problems. In this paper, we propose a new framework to adapt LLMs to specific domains and build Lawyer LLaMA, a legal domain LLM, based on this framework. Specifically, we inject domain knowledge during the continual training stage and teach the model to learn professional skills using properly designed supervised fine-tuning tasks. Moreover, to alleviate the hallucination problem during the model's generation, we add a retrieval module and extract relevant legal articles before the model answers any queries. When learning domain-specific skills, we find that experts' experience is much more useful than experiences distilled from ChatGPT, where hundreds of expert-written data outperform tens of thousands of ChatGPT-generated ones. We will release our model and data.