# BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

arXiv: https://arxiv.org/abs/2605.28183 · HTML: https://mlsanity.com/paper/2605.28183/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 6 of its 36 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 6 of 6)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain](https://mlsanity.com/paper/2408.10343/) (2024)
- [LEXam: Benchmarking Legal Reasoning on 340 Law Exams](https://mlsanity.com/paper/2505.12864/) (2025)
- [AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark](https://mlsanity.com/paper/2508.19365/) (2025)
- [BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks](https://mlsanity.com/paper/2604.13583/) (2026)
- [Legal RAG Bench: an end-to-end benchmark for legal RAG](https://mlsanity.com/paper/2603.01710/) (2026)

## Abstract (arXiv)

We introduce BenGER (Benchmark for German Law), a benchmark and dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The dataset combines 596 exam-style free-text legal case tasks across multiple levels of legal education and 531 short doctrinal reasoning tasks. It includes a controlled validation subset of timed human-written solutions under both unaided and human-AI co-creation conditions. We evaluate 12 contemporary LLM systems - closed flagship, efficiency-oriented, and open-weight - with a rubric-aligned LLM-as-a-Judge cross-validated against a multi-rater human-grading layer (three blind reviews per solution, six judge families benchmarked against the human pool). Closed-flagship systems lead the leaderboard across all three corpora, human-AI co-creation measurably improves on unaided human work, and the LLM judge tracks human grading at Pearson r=0.76 and Cohen's k=0.60. System rankings are stable across judge families and two judges from independent providers clear the Calderon single-reviewer replacement bar on human-authored solutions.