mlsanityData to 7 Oct 2026

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

arXiv 2605.28183 ↗First posted 27 May 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

We introduce BenGER (Benchmark for German Law), a benchmark and dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The dataset combines 596 exam-style free-text legal case tasks across multiple levels of legal education and 531 short doctrinal reasoning tasks. It includes a controlled validation subset of timed human-written solutions under both unaided and human-AI co-creation conditions. We evaluate 12 contemporary LLM systems - closed flagship, efficiency-oriented, and open-weight - with a rubric-aligned LLM-as-a-Judge cross-validated against a multi-rater human-grading layer (three blind reviews per solution, six judge families benchmarked against the human pool). Closed-flagship systems lead the leaderboard across all three corpora, human-AI co-creation measurably improves on unaided human work, and the LLM judge tracks human grading at Pearson r=0.76 and Cohen's k=0.60. System rankings are stable across judge families and two judges from independent providers clear the Calderon single-reviewer replacement bar on human-authored solutions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 6 of its 36 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

6 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks15 Apr
  2. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Exams19 May

Other references · 4

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 113
  3. 2025AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmarkcited by 4
  4. 2026Legal RAG Bench: an end-to-end benchmark for legal RAGcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  2. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
  3. 2025LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation
  4. 2026GradeLegal: Automated Grading for German Legal Cases
  5. 2025A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences
  6. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domain
  7. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice
  8. 2025Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond