mlsanityData to 14 Oct 2026

GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations

arXiv 2505.17267 ↗First posted 22 May 2025cs.CLcited by 15

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

We introduce GreekBarBench, a benchmark that evaluates LLMs on legal questions across five different legal areas from the Greek Bar exams, requiring citations to statutory articles and case facts. To tackle the challenges of free-text evaluation, we propose a three-dimensional scoring system combined with an LLM-as-a-judge approach. We also develop a meta-evaluation benchmark to assess the correlation between LLM-judges and human expert evaluations, revealing that simple, span-based rubrics improve their alignment. Our systematic evaluation of 13 proprietary and open-weight LLMs shows that even though the best models outperform average expert scores, they fall short of the 95th percentile of experts.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 13 of its 45 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

13 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025LEXam: Benchmarking Legal Reasoning on 340 Law Exams19 May

Other references · 12

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2025Gemma 3 Technical Reportcited by 1,824
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 321
  5. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  6. 2024Better Call GPT, Comparing Large Language Models Against Lawyerscited by 26
  7. 2025Plutus: Benchmarking Large Language Models in Low-Resource Greek Financecited by 18
  8. 2024Meltemi: The first open Large Language Model for Greekcited by 17
  9. 2025Krikri: Advancing Open Large Language Models for Greekcited by 14
  10. 2024Evaluating AI for Law: Bridging the Gap with Open-Source Solutionscited by 11
  11. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 11
  12. 2025CaseGen: A Benchmark for Multi-Stage Legal Case Documents Generationcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 13.

15 papers.

Sort:

By area:

  1. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 32
  2. 2026GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greekcited by 6
  3. 2025LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Textcited by 3
  4. 2026Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmarkcited by 2
  5. 2026Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Taskscited by 2
  6. 2026AppellateGen: A Benchmark for Appellate Legal Judgment Generationcited by 1
  7. 2026Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysiscited by 1
  8. 2026GreekBarRetrieval: A Benchmark for Greek Statutory Retrievalcited by 1
  9. 2026GradeLegal: Automated Grading for German Legal Casescited by 0
  10. 2026Reference-Free Evaluation of Reasoning in Open-Ended Question Answeringcited by 0
  11. 2026A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Datasetcited by 0
  12. 2026Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responsescited by 0
3 more
  1. 2026Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmarkcited by 0
  2. 2026When Does Defendant Statement Matter? A Study of Bias and Persuasion in LLM-Simulated Jurorscited by 0
  3. 2026Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation
  2. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice
  3. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  4. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions
  5. 2026BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
  6. 2023LawBench: Benchmarking Legal Knowledge of Large Language Models
  7. 2023One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support
  8. 2026Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task