# GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations

arXiv: https://arxiv.org/abs/2505.17267 · HTML: https://mlsanity.com/paper/2505.17267/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 13 of its 45 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 13 of 13)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks](https://mlsanity.com/paper/2406.18403/) (2024)
- [LEXam: Benchmarking Legal Reasoning on 340 Law Exams](https://mlsanity.com/paper/2505.12864/) (2025)
- [Better Call GPT, Comparing Large Language Models Against Lawyers](https://mlsanity.com/paper/2401.16212/) (2024)
- [Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance](https://mlsanity.com/paper/2502.18772/) (2025)
- [Meltemi: The first open Large Language Model for Greek](https://mlsanity.com/paper/2407.20743/) (2024)
- [Krikri: Advancing Open Large Language Models for Greek](https://mlsanity.com/paper/2505.13772/) (2025)
- [Evaluating AI for Law: Bridging the Gap with Open-Source Solutions](https://mlsanity.com/paper/2404.12349/) (2024)
- [SwiLTra-Bench: The Swiss Legal Translation Benchmark](https://mlsanity.com/paper/2503.01372/) (2025)
- [CaseGen: A Benchmark for Multi-Stage Legal Case Documents Generation](https://mlsanity.com/paper/2502.17943/) (2025)

## Built on by (most cited first, 15 of 15)

- [PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning](https://mlsanity.com/paper/2511.11562/) (2025)
- [GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek](https://mlsanity.com/paper/2602.05150/) (2026)
- [LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text](https://mlsanity.com/paper/2505.24826/) (2025)
- [Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark](https://mlsanity.com/paper/2602.16811/) (2026)
- [Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks](https://mlsanity.com/paper/2605.08437/) (2026)
- [AppellateGen: A Benchmark for Appellate Legal Judgment Generation](https://mlsanity.com/paper/2601.01331/) (2026)
- [Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis](https://mlsanity.com/paper/2608.15630/) (2026)
- [GreekBarRetrieval: A Benchmark for Greek Statutory Retrieval](https://mlsanity.com/paper/2608.18752/) (2026)
- [GradeLegal: Automated Grading for German Legal Cases](https://mlsanity.com/paper/2605.21076/) (2026)
- [Reference-Free Evaluation of Reasoning in Open-Ended Question Answering](https://mlsanity.com/paper/2607.19678/) (2026)
- [A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset](https://mlsanity.com/paper/2607.21274/) (2026)
- [Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses](https://mlsanity.com/paper/2608.17810/) (2026)
- [Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark](https://mlsanity.com/paper/2608.30327/) (2026)
- [When Does Defendant Statement Matter? A Study of Bias and Persuasion in LLM-Simulated Jurors](https://mlsanity.com/paper/2609.09887/) (2026)
- [Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks](https://mlsanity.com/paper/2609.34800/) (2026)

## Abstract (arXiv)

We introduce GreekBarBench, a benchmark that evaluates LLMs on legal questions across five different legal areas from the Greek Bar exams, requiring citations to statutory articles and case facts. To tackle the challenges of free-text evaluation, we propose a three-dimensional scoring system combined with an LLM-as-a-judge approach. We also develop a meta-evaluation benchmark to assess the correlation between LLM-judges and human expert evaluations, revealing that simple, span-based rubrics improve their alignment. Our systematic evaluation of 13 proprietary and open-weight LLMs shows that even though the best models outperform average expert scores, they fall short of the 95th percentile of experts.