# LAiW: A Chinese Legal Large Language Models Benchmark

arXiv: https://arxiv.org/abs/2310.05620 · HTML: https://mlsanity.com/paper/2310.05620/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 36 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 15 of 15)

- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence](https://mlsanity.com/paper/2004.12158/) (2020)
- [Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture](https://mlsanity.com/paper/2306.16092/) (2023)
- [DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services](https://mlsanity.com/paper/2309.11325/) (2023)
- [A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges](https://mlsanity.com/paper/2204.04859/) (2022)
- [CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain](https://mlsanity.com/paper/1911.08962/) (2019)
- [CJRC: A Reliable Human-Annotated Benchmark DataSet for Chinese Judicial Reading Comprehension](https://mlsanity.com/paper/1912.09156/) (2019)
- [Legal Judgment Prediction with Multi-Stage CaseRepresentation Learning in the Real Court Setting](https://mlsanity.com/paper/2107.05192/) (2021)
- [Learning Fine-grained Fact-Article Correspondence in Legal Cases](https://mlsanity.com/paper/2104.10726/) (2021)
- [Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?](https://mlsanity.com/paper/2308.06032/) (2023)

## Built on by (most cited first, 9 of 9)

- [LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2409.20288/) (2024)
- [LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Model](https://mlsanity.com/paper/2406.04614/) (2024)
- [PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation](https://mlsanity.com/paper/2407.05721/) (2024)
- [CitaLaw: Enhancing LLM with Citations in Legal Domain](https://mlsanity.com/paper/2412.14556/) (2024)
- [TransportationGames: Benchmarking Transportation Knowledge of (Multimodal) Large Language Models](https://mlsanity.com/paper/2401.04471/) (2024)
- [CaseGen: A Benchmark for Multi-Stage Legal Case Documents Generation](https://mlsanity.com/paper/2502.17943/) (2025)
- [Enterprise Benchmarks for Large Language Model Evaluation](https://mlsanity.com/paper/2410.12857/) (2024)
- [CPSDBench: A Large Language Model Evaluation Benchmark and Baseline for Chinese Public Security Domain](https://mlsanity.com/paper/2402.07234/) (2024)
- [Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/2601.15267/) (2026)

## Abstract (arXiv)

General and legal domain LLMs have demonstrated strong performance in various tasks of LegalAI. However, the current evaluations of these LLMs in LegalAI are defined by the experts of computer science, lacking consistency with the logic of legal practice, making it difficult to judge their practical capabilities. To address this challenge, we are the first to build the Chinese legal LLMs benchmark LAiW, based on the logic of legal practice. To align with the thinking process of legal experts and legal practice (syllogism), we divide the legal capabilities of LLMs from easy to difficult into three levels: basic information retrieval, legal foundation inference, and complex legal application. Each level contains multiple tasks to ensure a comprehensive evaluation. Through automated evaluation of current general and legal domain LLMs on our benchmark, we indicate that these LLMs may not align with the logic of legal practice. LLMs seem to be able to directly acquire complex legal application capabilities but perform poorly in some basic tasks, which may pose obstacles to their practical application and acceptance by legal experts. To further confirm the complex legal application capabilities of current LLMs in legal application scenarios, we also incorporate human evaluation with legal experts. The results indicate that while LLMs may demonstrate strong performance, they still require reinforcement of legal logic.