# Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks

arXiv: https://arxiv.org/abs/2605.08437 · HTML: https://mlsanity.com/paper/2605.08437/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 6 of 6)

- [LEXam: Benchmarking Legal Reasoning on 340 Law Exams](https://mlsanity.com/paper/2505.12864/) (2025)
- [JuDGE: Benchmarking Judgment Document Generation for Chinese Legal System](https://mlsanity.com/paper/2503.14258/) (2025)
- [GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations](https://mlsanity.com/paper/2505.17267/) (2025)
- [PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice](https://mlsanity.com/paper/2601.16669/) (2026)
- [Automatic Legal Writing Evaluation of LLMs](https://mlsanity.com/paper/2504.21202/) (2025)
- [Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities](https://mlsanity.com/paper/2512.24572/) (2025)

## Built on by (most cited first, 2 of 2)

- [JusticeAxis: Benchmarking Legal Judgment between Rigid Rule Application and Ungrounded Discretion](https://mlsanity.com/paper/2610.00353/) (2026)
- [SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges](https://mlsanity.com/paper/2610.19992/) (2026)

## Abstract (arXiv)

Existing benchmarks for legal AI focus primarily on tasks where LLMs must produce legal arguments or documents, yet the capacity to judge such arguments -- weighing competing claims, applying doctrine to facts, and rendering reasoned decisions -- is arguably as fundamental to a well-functioning legal system as advocacy itself. We introduce Magis-Bench, a benchmark for evaluating LLMs on magistrate-level writing tasks derived from recent Brazilian competitive examinations for judicial positions. Magis-Bench comprises 74 questions from eight examinations conducted between 2023 and 2025, including discursive legal analysis questions with multi-turn structure and practical exercises requiring the composition of complete civil and criminal judicial sentences. We evaluate 23 state-of-the-art LLMs using an LLM-as-a-judge methodology with four independent frontier models as evaluators. Our results show strong inter-judge agreement (Kendall's W = 0.984; pairwise Kendall's τ ≥ 0.897), with Google's Gemini-3-Pro-Preview achieving the highest average score (6.97/10), followed by Gemini-3-Flash-Preview (6.67) and Claude-4.5-Opus (6.46). Even the best-performing models score below 70% of the maximum, indicating that judicial-level legal reasoning and writing remain challenging for current LLMs. We release the complete benchmark, model outputs, and evaluation code to support further research on legal AI capabilities.