# Supporting Humans in Evaluating AI Summaries of Legal Depositions

arXiv: https://arxiv.org/abs/2601.15182 · HTML: https://mlsanity.com/paper/2601.15182/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Builds on (most cited first, 7 of 7)

- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework](https://mlsanity.com/paper/2411.09607/) (2024)
- [The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models](https://mlsanity.com/paper/2504.15068/) (2025)
- [On the Evaluation of Machine-Generated Reports](https://mlsanity.com/paper/2405.00982/) (2024)
- [Auto-ARGUE: LLM-Based Report Generation Evaluation](https://mlsanity.com/paper/2509.26184/) (2025)
- [Question-Answering Approach to Evaluating Legal Summaries](https://mlsanity.com/paper/2309.15016/) (2023)
- [Criteria-Based LLM Relevance Judgments](https://mlsanity.com/paper/2507.09488/) (2025)

## Abstract (arXiv)

While large language models (LLMs) are increasingly used to summarize long documents, this trend poses significant challenges in the legal domain, where the factual accuracy of deposition summaries is crucial. Nugget-based methods have been shown to be extremely helpful for the automated evaluation of summarization approaches. In this work, we translate these methods to the user side and explore how nuggets could directly assist end users. Although prior systems have demonstrated the promise of nugget-based evaluation, its potential to support end users remains underexplored. Focusing on the legal domain, we present a prototype that leverages a factual nugget-based approach to support legal professionals in two concrete scenarios: (1) determining which of two summaries is better, and (2) manually improving an automatically generated summary.