mlsanityData to 7 Oct 2026

LEXam: Benchmarking Legal Reasoning on 340 Law Exams

arXiv 2505.12864 ↗First posted 19 May 2025cs.CL cs.AI cs.LGcited by 52

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Long-form legal reasoning remains a key challenge for large language models (LLMs) in spite of recent advances in test-time scaling. To address this, we introduce LEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels. The dataset comprises 7,537 law exam questions in English and German. It includes both long-form, open-ended questions and multiple-choice questions with varying numbers of options. Besides reference answers, the open questions are also accompanied by explicit guidance outlining the expected legal reasoning approach such as issue spotting, rule recall, or rule application. Our evaluation on both open-ended and multiple-choice questions present significant challenges for current LLMs; in particular, they notably struggle with open questions that require structured, multi-step legal reasoning. Moreover, our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities. Deploying an ensemble LLM-as-a-Judge paradigm with rigorous human expert validation, we demonstrate how model-generated reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments. Our evaluation setup provides a scalable method to assess legal reasoning quality beyond simple accuracy metrics. Project page: https://lexam-benchmark.github.io/.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 34 of its 97 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

34 of 97 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LAR-ECHR: A New Legal Argument Reasoning Task and Dataset for Cases of the European Court of Human Rights17 Oct
  2. 2023Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement27 Oct
  3. 2023One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support15 Jun

Other references · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2025Gemma 3 Technical Reportcited by 1,686
  9. 2024DeepSeek-V3 Technical Reportcited by 724
  10. 2024Phi-4 Technical Reportcited by 675
  11. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
  12. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
19 more
  1. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  2. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  3. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  4. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
  5. 2025PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoningcited by 58
  6. 2023FIMO: A Challenge Formal Dataset for Automated Theorem Provingcited by 54
  7. 2023Explaining Legal Concepts with Augmented Large Language Models (GPT-4)cited by 39
  8. 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 36
  9. 2025PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solvingcited by 31
  10. 2023A Comprehensive Evaluation of Large Language Models on Legal Judgment Predictioncited by 27
  11. 2024Leveraging Large Language Models for Relevance Judgments in Legal Case Retrievalcited by 16
  12. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 10
  13. 2023Legal Extractive Summarization of U.S. Court Opinionscited by 9
  14. 2023The Law and NLP: Bridging Disciplinary Disconnectscited by 8
  15. 2024Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization in Switzerlandcited by 6
  16. 2025Automating Legal Interpretation with LLMs: Retrieval, Generation, and Evaluationcited by 5
  17. 2023LePaRD: A Large-Scale Dataset of Judges Citing Precedentscited by 3
  18. 2025LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streamscited by 2
  19. 2024From Citations to Criticality: Predicting Legal Decision Influence in the Multilingual Swiss Jurisprudencecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 41.

52 papers.

Sort:

By area:

  1. 2025Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoningcited by 17
  2. 2025GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citationscited by 15
  3. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 13
  4. 2025L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness Auditcited by 11
  5. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?cited by 9
  6. 2026Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Modelscited by 7
  7. 2025CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfittingcited by 6
  8. 2026Reasoning-Driven Synthetic Data Generation and Evaluationcited by 4
  9. 2025When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governancecited by 3
  10. 2025Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarizationcited by 3
  11. 2025Evaluating the Limits of Large Language Models in Multilingual Legal Reasoningcited by 3
  12. 2026How Reliable are Confidence Estimators for Large Reasoning Models? A Systematic Benchmark on High-Stakes Domainscited by 3
38 more
  1. 2026OpenExempt: A Diagnostic Benchmark for Legal Reasoning and a Framework for Creating Custom Benchmarks on Demandcited by 3
  2. 2025KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialoguescited by 2
  3. 2025Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilitiescited by 2
  4. 2026EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economiescited by 2
  5. 2026HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Modelscited by 2
  6. 2026GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluationcited by 2
  7. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  8. 2026Evaluation of Large Language Models in Legal Applications: Challenges, Methods, and Future Directionscited by 1
  9. 2026Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Taskscited by 1
  10. 2026Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Taskscited by 1
  11. 2026Beyond Imperfect Alternatives with Rulemapping: A Neuro-Symbolic Case Study on Online Hate Speechcited by 1
  12. 2026Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditionscited by 1
  13. 2026From Statute to Control Flow: Span-Grounded Deontic Trees for Defeasible Scope Parsingcited by 1
  14. 2026CrossLex: A Source-Grounded Benchmark for Cross-Jurisdictional Legal Reasoning in Large Language Modelscited by 1
  15. 2026Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarkscited by 1
  16. 2025Evaluating the Role of Verifiers in Test-Time Scaling for Legal Reasoning Taskscited by 0
  17. 2026Investigating Disability Representations in Text-to-Image Modelscited by 0
  18. 2026CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Lawcited by 0
  19. 2026NyayaMind- A Framework for Transparent Legal Reasoning and Judgment Prediction in the Indian Legal Systemcited by 0
  20. 2026Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimizationcited by 0
  21. 2026Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Taskcited by 0
  22. 2026LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Modelscited by 0
  23. 2026Accurate Legal Reasoning at Scale: Neuro-Symbolic Offloading and Structural Auditability for Robust Legal Adjudicationcited by 0
  24. 2026TriBench-Ko: Evaluating LLM Risks in Judicial Workflowscited by 0
  25. 2026GradeLegal: Automated Grading for German Legal Casescited by 0
  26. 2026Decompose-and-Refine: Structured Legal Question Answering with Parametric Retrievalcited by 0
  27. 2026BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Lawcited by 0
  28. 2026Quantum-Inspired Trace-Augmented Evidence Selection for Reasoning over Structured Hypothesis Spacescited by 0
  29. 2026Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courtscited by 0
  30. 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Modelscited by 0
  31. 2026Investigating Multi-Agent Deliberation in Lawcited by 0
  32. 2026L-MAD: A Systematic Evaluation of Multi-Agent Debate Structures in Legal Reasoningcited by 0
  33. 2026Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Lawcited by 0
  34. 2026Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneckcited by 0
  35. 2026ContractScrub: A benchmark for final review of legal contractscited by 0
  36. 2026InsufficiencyBench: Evaluating LLM legal advice on underspecified user queriescited by 0
  37. 2026Building Legal Reward Models for Grounding and Abstentioncited by 0
  38. 2026How Much is a Human Right Worth? ECtHR-NPD: A Benchmark for Predicting Non-Pecuniary Damage Awardscited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Llama walks into the 'Bar': Efficient Supervised Fine-Tuning for Legal Reasoning in the Multi-state Bar Exam
  2. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  3. 2025LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation
  4. 2025Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning
  5. 2025Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond
  6. 2025Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning
  7. 2025Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models
  8. 2025LegalBench.PT: A Benchmark for Portuguese Law