# Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions arXiv: https://arxiv.org/abs/2605.29738 ยท HTML: https://mlsanity.com/paper/2605.29738/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. - Builds on 20 of its 40 references that are in the map. - First posted on arXiv 28 May 2026. ## Placed in - LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/) ## Builds on (most cited first, 20 of 20) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023) - [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021) - [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021) - [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023) - [MultiLegalPile: A 689GB Multilingual Legal Corpus](https://mlsanity.com/paper/2306.02069/) (2023) - [LEXam: Benchmarking Legal Reasoning on 340 Law Exams](https://mlsanity.com/paper/2505.12864/) (2025) - [The Amazon Nova Family of Models: Technical Report and Model Card](https://mlsanity.com/paper/2506.12103/) (2025) - [Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2604.12374/) (2026) - [NVIDIA Nemotron Nano V2 VL](https://mlsanity.com/paper/2511.03929/) (2025) - [An Empirical Study on Cross-X Transfer for Legal Judgment Prediction](https://mlsanity.com/paper/2209.12325/) (2022) - [One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support](https://mlsanity.com/paper/2306.09237/) (2023) - [ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models](https://mlsanity.com/paper/2408.07983/) (2024) - [The Tokenizer Tax Across 25 European Languages: Domain Invariance, Cross-Lingual Few-Shot Effects, and the Ukrainian Penalty](https://mlsanity.com/paper/2605.24718/) (2026) - [Evaluating the Limits of Large Language Models in Multilingual Legal Reasoning](https://mlsanity.com/paper/2509.22472/) (2025) - [LegalBench.PT: A Benchmark for Portuguese Law](https://mlsanity.com/paper/2502.16357/) (2025) - [LegalRikai: Open Benchmark -- Benchmark for Complex Japanese Corporate Legal Tasks](https://mlsanity.com/paper/2512.11297/) (2025) - [VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models](https://mlsanity.com/paper/2512.14554/) (2025) - [UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning](https://mlsanity.com/paper/2605.29170/) (2026) ## Built on by (most cited first, 1 of 1) - [CrossLex: A Source-Grounded Benchmark for Cross-Jurisdictional Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2608.01292/) (2026) ## Abstract (arXiv) Court registries publish millions of decisions with structured metadata, which makes them an attractive source of labelled legal benchmarks: the court type, the form of the decision and its subject area come for free. We show that this convenience has a cost that registry-derived benchmarks rarely measure. We build Multi-Legal-Bench, which evaluates identical tasks on native court decisions from five national registries (France, the Netherlands, Poland, the Czech Republic and Lithuania), extending the Ukrainian UA-Legal-Bench, and audit what its cells actually measure. A keyword scan that uses no model reaches 96% on Dutch judgment-form classification against a 50% majority baseline, 84% on French court-type classification against 33%, and 66% on Polish judgment-form classification, where all nine models add at most seven points over it. Replacing the label names in the input by a mask brings the scan to the majority floor in all six affected cells, and eight of nine models lose accuracy significantly in most of them (39 of 54 model-cell pairs), by up to 48 points (Dutch judgment form: 100% to 52%); only Claude Sonnet 5 is essentially unaffected. Paired McNemar tests with Holm correction find a leader that beats every other model in only two of twelve cells, each a different model; but once the label names are masked, the same model leads both, and the spread between models widens in every masked cell. Part of the apparent parity between models is produced by the leakage itself. The audit also exposed a scoring defect in our own earlier release (answers written with diacritics were scored as wrong, understating Czech judgment-form accuracy by up to 90 points), which we correct and document. We recommend that every benchmark built from court registries report a no-model baseline and a masked control per cell. All data, prompts, predictions and scoring code are released.