# Improved Evidence Extraction and Metrics for Document Inconsistency Detection with LLMs

arXiv: https://arxiv.org/abs/2601.02627 · HTML: https://mlsanity.com/paper/2601.02627/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 5 of 5)

- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [DocNLI: A Large-scale Dataset for Document-level Natural Language Inference](https://mlsanity.com/paper/2106.09449/) (2021)
- [Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters](https://mlsanity.com/paper/2204.07447/) (2022)
- [Improved LLM Agents for Financial Document Question Answering](https://mlsanity.com/paper/2506.08726/) (2025)

## Abstract (arXiv)

Large language models (LLMs) are becoming useful in many domains due to their impressive abilities that arise from large training datasets and large model sizes. However, research on LLM-based approaches to document inconsistency detection is relatively limited. We address this gap by investigating evidence extraction capabilties of LLMs for document inconsistency detection. To this end, we introduce new comprehensive evidence-extraction metrics and a redact-and-retry framework with constrained filtering that substantially improves evidence extraction performance over other prompting methods. We support our approach with strong experimental results and release a new semi-synthetic dataset for evaluating evidence extraction.