# Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directions

arXiv: https://arxiv.org/abs/2601.15267 · HTML: https://mlsanity.com/paper/2601.15267/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 95 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Builds on (most cited first, 30 of 51)

- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021)
- [BillSum: A Corpus for Automatic Summarization of US Legislation](https://mlsanity.com/paper/1910.00523/) (2019)
- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [JEC-QA: A Legal-Domain Question Answering Dataset](https://mlsanity.com/paper/1911.12011/) (2019)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [Question Answering for Privacy Policies: Combining Computational and Legal Perspectives](https://mlsanity.com/paper/1911.00841/) (2019)
- [A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering](https://mlsanity.com/paper/2005.05257/) (2020)
- [Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark](https://mlsanity.com/paper/2110.00806/) (2021)
- [LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development](https://mlsanity.com/paper/2305.07507/) (2023)
- [LEXam: Benchmarking Legal Reasoning on 340 Law Exams](https://mlsanity.com/paper/2505.12864/) (2025)
- [Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration](https://mlsanity.com/paper/2310.09241/) (2023)
- [FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing](https://mlsanity.com/paper/2203.07228/) (2022)
- [Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities](https://mlsanity.com/paper/2206.10883/) (2022)
- [A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction](https://mlsanity.com/paper/2206.05224/) (2022)
- [EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain](https://mlsanity.com/paper/2210.13448/) (2022)
- [LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Dataset](https://mlsanity.com/paper/2310.17609/) (2023)
- [AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents](https://mlsanity.com/paper/2408.08089/) (2024)
- [LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2409.20288/) (2024)
- [LEVEN: A Large-Scale Chinese Legal Event Detection Dataset](https://mlsanity.com/paper/2203.08556/) (2022)
- [Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2309.17050/) (2023)
- [MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding](https://mlsanity.com/paper/2301.00876/) (2023)
- [PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning](https://mlsanity.com/paper/2511.11562/) (2025)
- [Plain English Summarization of Contracts](https://mlsanity.com/paper/1906.00424/) (2019)

## Built on by (most cited first, 1 of 1)

- [LegalOne: A Family of Foundation Models for Reliable Legal Reasoning](https://mlsanity.com/paper/2602.00642/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are being increasingly integrated into legal applications, including judicial decision support, legal practice assistance, and public-facing legal services. While LLMs show strong potential in handling legal knowledge and tasks, their deployment in real-world legal settings raises critical concerns beyond surface-level accuracy, involving the soundness of legal reasoning processes and trustworthy issues such as fairness and reliability. Systematic evaluation of LLM performance in legal tasks has therefore become essential for their responsible adoption. This survey identifies key challenges in evaluating LLMs for legal tasks grounded in real-world legal practice. We analyze the major difficulties involved in assessing LLM performance in the legal domain, including outcome correctness, reasoning reliability, and trustworthiness. Building on these challenges, we review and categorize existing evaluation methods and benchmarks according to their task design, datasets, and evaluation metrics. We further discuss the extent to which current approaches address these challenges, highlight their limitations, and outline future research directions toward more realistic, reliable, and legally grounded evaluation frameworks for LLMs in legal domains.