# JuDGE: Benchmarking Judgment Document Generation for Chinese Legal System

arXiv: https://arxiv.org/abs/2503.14258 · HTML: https://mlsanity.com/paper/2503.14258/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- Retrieval-augmented generation / [Legal and financial RAG](https://mlsanity.com/rag/legal-and-financial-rag/)

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Optimizing Dense Retrieval Model Training with Hard Negatives](https://mlsanity.com/paper/2104.08051/) (2021)
- [DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services](https://mlsanity.com/paper/2309.11325/) (2023)
- [SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval](https://mlsanity.com/paper/2304.11370/) (2023)
- [Learning to Predict Charges for Criminal Cases with Legal Basis](https://mlsanity.com/paper/1707.09168/) (2017)
- [LawLLM: Law Large Language Model for the US Legal System](https://mlsanity.com/paper/2407.21065/) (2024)
- [Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2309.17050/) (2023)
- [Charge-Based Prison Term Prediction with Deep Gating Network](https://mlsanity.com/paper/1908.11521/) (2019)
- [RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects](https://mlsanity.com/paper/2501.18365/) (2025)
- [Knowledge Editing through Chain-of-Thought](https://mlsanity.com/paper/2412.17727/) (2024)
- [Yes, BM25 is a Strong Baseline for Legal Case Retrieval](https://mlsanity.com/paper/2105.05686/) (2021)
- [Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.00175/) (2024)
- [Mitigating Entity-Level Hallucination in Large Language Models](https://mlsanity.com/paper/2407.09417/) (2024)
- [THUIR@COLIEE 2023: Incorporating Structural Knowledge into Pre-trained Language Models for Legal Case Retrieval](https://mlsanity.com/paper/2305.06812/) (2023)
- [Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-hoc Retrieval](https://mlsanity.com/paper/2312.10661/) (2023)
- [THUIR2 at NTCIR-16 Session Search (SS) Task](https://mlsanity.com/paper/2307.00250/) (2023)
- [LeKUBE: A Legal Knowledge Update BEnchmark](https://mlsanity.com/paper/2407.14192/) (2024)
- [THUIR@COLIEE 2023: More Parameters and Legal Knowledge for Legal Case Entailment](https://mlsanity.com/paper/2305.06817/) (2023)
- [Rethinking Legal Judgement Prediction in a Realistic Scenario in the Era of Large Language Models](https://mlsanity.com/paper/2410.10542/) (2024)
- [Legal Question Answering using Ranking SVM and Deep Convolutional Neural Network](https://mlsanity.com/paper/1703.05320/) (2017)
- [Web Search via an Efficient and Effective Brain-Machine Interface](https://mlsanity.com/paper/2110.07225/) (2021)
- [Creating Auxiliary Representations from Charge Definitions for Criminal Charge Prediction](https://mlsanity.com/paper/1911.05202/) (2019)

## Built on by (most cited first, 18 of 18)

- [MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems](https://mlsanity.com/paper/2510.17281/) (2025)
- [Skill Retrieval Augmentation for Agentic AI](https://mlsanity.com/paper/2604.24594/) (2026)
- [SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation](https://mlsanity.com/paper/2508.15658/) (2025)
- [Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Models](https://mlsanity.com/paper/2506.04832/) (2025)
- [Decoupling Reasoning and Knowledge Injection for In-Context Knowledge Editing](https://mlsanity.com/paper/2506.00536/) (2025)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [Enhancing Judgment Document Generation via Agentic Legal Information Collection and Rubric-Guided Optimization](https://mlsanity.com/paper/2605.02011/) (2026)
- [LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence](https://mlsanity.com/paper/2512.04578/) (2025)
- [Dynamic and Parametric Retrieval-Augmented Generation](https://mlsanity.com/paper/2506.06704/) (2025)
- [Towards Unification of Hallucination Detection and Fact Verification for Large Language Models](https://mlsanity.com/paper/2512.02772/) (2025)
- [Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks](https://mlsanity.com/paper/2605.08437/) (2026)
- [CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation](https://mlsanity.com/paper/2602.12639/) (2026)
- [TrafficRAG: A Multimodal RAG Framework for Traffic Accident Liability Determination](https://mlsanity.com/paper/2606.01737/) (2026)
- [PL-CA: A Parametric Legal Case Augmentation Framework](https://mlsanity.com/paper/2509.06356/) (2025)
- [Na Pr'atica, qual IA Entende o Direito? Um Estudo Experimental com IAs Generalistas e uma IA Jur'idica](https://mlsanity.com/paper/2510.18108/) (2025)
- [RWGBench: Evaluating Scholarly Positioning in Related Work Generation](https://mlsanity.com/paper/2606.24894/) (2026)
- [Generative Chinese Statute Retrieval](https://mlsanity.com/paper/2607.11109/) (2026)
- [LexIssue: Benchmarking Legal Issue Identification in Chinese Civil Litigation](https://mlsanity.com/paper/2609.02954/) (2026)

## Abstract (arXiv)

This paper introduces JuDGE (Judgment Document Generation Evaluation), a novel benchmark for evaluating the performance of judgment document generation in the Chinese legal system. We define the task as generating a complete legal judgment document from the given factual description of the case. To facilitate this benchmark, we construct a comprehensive dataset consisting of factual descriptions from real legal cases, paired with their corresponding full judgment documents, which serve as the ground truth for evaluating the quality of generated documents. This dataset is further augmented by two external legal corpora that provide additional legal knowledge for the task: one comprising statutes and regulations, and the other consisting of a large collection of past judgment documents. In collaboration with legal professionals, we establish a comprehensive automated evaluation framework to assess the quality of generated judgment documents across various dimensions. We evaluate various baseline approaches, including few-shot in-context learning, fine-tuning, and a multi-source retrieval-augmented generation (RAG) approach, using both general and legal-domain LLMs. The experimental results demonstrate that, while RAG approaches can effectively improve performance in this task, there is still substantial room for further improvement. All the codes and datasets are available at: https://github.com/oneal2000/JuDGE.