# Evaluation Ethics of LLMs in Legal Domain

arXiv: https://arxiv.org/abs/2403.11152 · HTML: https://mlsanity.com/paper/2403.11152/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 18 references that are in the map.
- First posted on arXiv 17 Mar 2024.

## Builds on (most cited first, 4 of 4)

- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture](https://mlsanity.com/paper/2306.16092/) (2023)

## Built on by (most cited first, 9 of 9)

- [CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges](https://mlsanity.com/paper/2410.15393/) (2024)
- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation](https://mlsanity.com/paper/2502.20640/) (2025)
- [Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses](https://mlsanity.com/paper/2509.16093/) (2025)
- [LLM Agents in Law: Taxonomy, Applications, and Challenges](https://mlsanity.com/paper/2601.06216/) (2026)
- [LegalOne: A Family of Foundation Models for Reliable Legal Reasoning](https://mlsanity.com/paper/2602.00642/) (2026)
- [Logicbreaks: A Framework for Understanding Subversion of Rule-based Inference](https://mlsanity.com/paper/2407.00075/) (2024)
- [Automated Harmfulness Testing for Code Large Language Models](https://mlsanity.com/paper/2503.16740/) (2025)
- [Ethics Testing: Proactive Identification of Generative AI System Harms](https://mlsanity.com/paper/2604.22089/) (2026)

## Abstract (arXiv)

In recent years, the utilization of large language models for natural language dialogue has gained momentum, leading to their widespread adoption across various domains. However, their universal competence in addressing challenges specific to specialized fields such as law remains a subject of scrutiny. The incorporation of legal ethics into the model has been overlooked by researchers. We asserts that rigorous ethic evaluation is essential to ensure the effective integration of large language models in legal domains, emphasizing the need to assess domain-specific proficiency and domain-specific ethic. To address this, we propose a novelty evaluation methodology, utilizing authentic legal cases to evaluate the fundamental language abilities, specialized legal knowledge and legal robustness of large language models (LLMs). The findings from our comprehensive evaluation contribute significantly to the academic discourse surrounding the suitability and performance of large language models in legal domains.