# Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics

arXiv: https://arxiv.org/abs/2512.01020 · HTML: https://mlsanity.com/paper/2512.01020/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 29 of its 64 references that are in the map.
- First posted on arXiv 30 Nov 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 29 of 29)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains](https://mlsanity.com/paper/2507.17746/) (2025)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval](https://mlsanity.com/paper/2407.12883/) (2024)
- [Checklists Are Better Than Reward Models For Aligning Language Models](https://mlsanity.com/paper/2507.18624/) (2025)
- [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024)
- [The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models](https://mlsanity.com/paper/2406.05761/) (2024)
- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [ReasonIR: Training Retrievers for Reasoning Tasks](https://mlsanity.com/paper/2504.20595/) (2025)
- [Evaluating Step-by-step Reasoning Traces: A Survey](https://mlsanity.com/paper/2502.12289/) (2025)
- [EXAONE 3.5: Series of Large Language Models for Real-world Use Cases](https://mlsanity.com/paper/2412.04862/) (2024)
- [A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges](https://mlsanity.com/paper/2204.04859/) (2022)
- [Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction](https://mlsanity.com/paper/2307.08321/) (2023)
- [Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics](https://mlsanity.com/paper/2503.23989/) (2025)
- [Yes, BM25 is a Strong Baseline for Legal Case Retrieval](https://mlsanity.com/paper/2105.05686/) (2021)
- [Scaling Evaluation-time Compute with Reasoning Models as Evaluators](https://mlsanity.com/paper/2503.19877/) (2025)
- [LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset](https://mlsanity.com/paper/2403.04791/) (2024)
- [Process-Supervised Reward Models for Verifying Clinical Note Generation: A Scalable Approach Guided by Domain Expertise](https://mlsanity.com/paper/2412.12583/) (2024)
- [LegalSearchLM: Rethinking Legal Case Retrieval as Legal Elements Generation](https://mlsanity.com/paper/2505.23832/) (2025)

## Built on by (most cited first, 5 of 5)

- [PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice](https://mlsanity.com/paper/2601.16669/) (2026)
- [From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape](https://mlsanity.com/paper/2606.08625/) (2026)
- [JPO: Juris Policy Optimization for Structured Legal Reasoning in Criminal Judgment Prediction](https://mlsanity.com/paper/2608.29616/) (2026)
- [FinancialAuditBench: Benchmark Construction under Differential Privacy Using Real-World Priors](https://mlsanity.com/paper/2609.32835/) (2026)
- [InvestigationWorlds: An Agentic Environment for Legal Investigation](https://mlsanity.com/paper/2610.04129/) (2026)

## Abstract (arXiv)

Evaluating the quality of LLM-generated reasoning traces in expert domains (e.g., law) is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such reasoning tasks. We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset with an emphasis on reasoning trace evaluation. We convert court judgments into hierarchical trees of opposing parties' arguments and the court's conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces. We verify the reliability of these rubrics via human expert annotations and comparison with coarse, less informative rubrics. Using the LEGIT dataset, we show that (1) LLMs' legal reasoning ability is seriously affected by both legal issue coverage and correctness, and that (2) retrieval-augmented generation (RAG) and RL with rubrics bring complementary benefits for legal reasoning abilities, where RAG improves overall reasoning capability, whereas RL improves correctness albeit with reduced coverage.