mlsanityData to 7 Oct 2026

Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics

arXiv 2512.01020 ↗First posted 30 Nov 2025cs.AI cs.CLcited by 4

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Evaluating the quality of LLM-generated reasoning traces in expert domains (e.g., law) is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such reasoning tasks. We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset with an emphasis on reasoning trace evaluation. We convert court judgments into hierarchical trees of opposing parties' arguments and the court's conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces. We verify the reliability of these rubrics via human expert annotations and comparison with coarse, less informative rubrics. Using the LEGIT dataset, we show that (1) LLMs' legal reasoning ability is seriously affected by both legal issue coverage and correctness, and that (2) retrieval-augmented generation (RAG) and RL with rubrics bring complementary benefits for legal reasoning abilities, where RAG improves overall reasoning capability, whereas RL improves correctness albeit with reduced coverage.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 29 of its 64 references that are in the map. First posted on arXiv 30 Nov 2025.

Builds on

29 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction17 Jul

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  9. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  10. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  12. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
16 more
  1. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  2. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  3. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  4. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  5. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  6. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  7. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  8. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  9. 2022A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challengescited by 50
  10. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 50
  11. 2025Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubricscited by 29
  12. 2021Yes, BM25 is a Strong Baseline for Legal Case Retrievalcited by 24
  13. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
  14. 2024LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Datasetcited by 8
  15. 2024Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertisecited by 5
  16. 2025LegalSearchLM: Rethinking Legal Case Retrieval as Legal Elements Generationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 13
  2. 2026From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscapecited by 4
  3. 2026JPO: Juris Policy Optimization for Structured Legal Reasoning in Criminal Judgment Predictioncited by 0
  4. 2026FinancialAuditBench: Benchmark Construction under Differential Privacy Using Real-World Priorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  2. 2025LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation
  3. 2025A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences
  4. 2026LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning
  5. 2025GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations
  6. 2026BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
  7. 2026LegalGraphRAG: Multi-Agent Graph Retrieval-Augmented Generation for Reliable Legal Reasoning
  8. 2026CrossLex: A Source-Grounded Benchmark for Cross-Jurisdictional Legal Reasoning in Large Language Models