# LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation

arXiv: https://arxiv.org/abs/2510.07243 · HTML: https://mlsanity.com/paper/2510.07243/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 13 of 13)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding](https://mlsanity.com/paper/2503.05061/) (2025)
- [MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding](https://mlsanity.com/paper/2301.00876/) (2023)
- [Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judge](https://mlsanity.com/paper/2410.03775/) (2024)
- [Insights on Disagreement Patterns in Multimodal Safety Perception across Diverse Rater Groups](https://mlsanity.com/paper/2410.17032/) (2024)
- [LalaEval: A Holistic Human Evaluation Framework for Domain-Specific Large Language Models](https://mlsanity.com/paper/2408.13338/) (2024)

## Built on by (most cited first, 10 of 10)

- [Efficient Evaluation of LLM Performance with Statistical Guarantees](https://mlsanity.com/paper/2601.20251/) (2026)
- [Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering](https://mlsanity.com/paper/2605.23497/) (2026)
- [Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring](https://mlsanity.com/paper/2605.24737/) (2026)
- [LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI](https://mlsanity.com/paper/2606.18021/) (2026)
- [Knowing the Form, Not the Function: Automatically Auditing Answer--Authority Decoupling in Legal Benchmarks](https://mlsanity.com/paper/2608.02621/) (2026)
- [LAMUS: A Large-Scale Corpus for Legal Argument Mining from U.S. Caselaw using LLMs](https://mlsanity.com/paper/2603.08286/) (2026)
- [Hallucinations in LLMs: A Lifecycle-Based Survey of Causes, Detection, Mitigation, and Prevention](https://mlsanity.com/paper/2608.26168/) (2026)
- [LexFlip: A Dissociation Diagnostic for Legal Meaning Preservation Metrics](https://mlsanity.com/paper/2609.05296/) (2026)
- [LexLattice: Multilingual Extractive Summarization via Neural Cellular Automata on Document Hierarchies](https://mlsanity.com/paper/2609.27032/) (2026)
- [SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges](https://mlsanity.com/paper/2610.19992/) (2026)

## Abstract (arXiv)

Evaluating large language model (LLM) outputs in the legal domain presents unique challenges due to the complex and nuanced nature of legal analysis. Current evaluation approaches either depend on reference data, which is costly to produce, or use standardized assessment methods, both of which have significant limitations for legal applications. Although LLM-as-a-Judge has emerged as a promising evaluation technique, its reliability and effectiveness in legal contexts depend heavily on evaluation processes unique to the legal industry and how trustworthy the evaluation appears to the human legal expert. This is where existing evaluation methods currently fail and exhibit considerable variability. This paper aims to close the gap: a) we break down lengthy responses into 'Legal Data Points' (LDPs), self-contained units of information, and introduce a novel, reference-free evaluation methodology that reflects how lawyers evaluate legal answers; b) we demonstrate that our method outperforms a variety of baselines on both our proprietary dataset and an open-source dataset (LegalBench); c) we show how our method correlates more closely with human expert evaluations and helps improve inter-annotator agreement; and finally d) we open source our Legal Data Points for a subset of LegalBench used in our experiments, allowing the research community to replicate our results and advance research in this vital area of LLM evaluation on legal question-answering.