mlsanityData to 7 Oct 2026

On Finding Inconsistencies in Documents

arXiv 2512.18601 ↗First posted 21 Dec 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Professionals in academia, law, and finance audit their documents because inconsistencies can result in monetary, reputational, and scientific costs. Language models (LMs) have the potential to dramatically speed up this auditing process. To understand their abilities, we introduce a benchmark, FIND (Finding INconsistencies in Documents), where each example is a document with an inconsistency inserted manually by a domain expert. Despite the documents being long, technical, and complex, the best-performing model (gpt-5) recovered 64% of the inserted inconsistencies. Surprisingly, gpt-5 also found undiscovered inconsistencies present in the original documents. For example, on 50 arXiv papers, we judged 136 out of 196 of the model's suggestions to be legitimate inconsistencies missed by the original authors. However, despite these findings, even the best models miss almost half of the inconsistencies in FIND, demonstrating that inconsistency detection is still a challenging task.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 61 references that are in the map. First posted on arXiv 21 Dec 2025.

Builds on

38 of 61 references.

References · 38

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  6. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  7. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2024Measuring short-form factuality in large language modelscited by 345
  10. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  11. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 148
  12. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 142
26 more
  1. 2024Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationscited by 130
  2. 2021ProoFVer: Natural Logic Theorem Proving for Fact Verificationcited by 52
  3. 2023Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noisecited by 52
  4. 2024WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipediacited by 43
  5. 2025Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?cited by 43
  6. 2023Data Portraits: Recording Foundation Model Training Datacited by 29
  7. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievalscited by 27
  8. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 25
  9. 2025DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMscited by 21
  10. 2024Who's Who: Large Language Models Meet Knowledge Conflicts in Practicecited by 20
  11. 2024ECon: On the Detection and Resolution of Evidence Conflictscited by 18
  12. 2024Open Domain Question Answering with Conflicting Contextscited by 18
  13. 2021WikiContradiction: Detecting Self-Contradiction Articles on Wikipediacited by 17
  14. 2025FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMscited by 14
  15. 2025Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluationcited by 12
  16. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 12
  17. 2024Prospects for inconsistency detection using large language models and sheavescited by 10
  18. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 10
  19. 2025LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documentscited by 7
  20. 2025Better Call CLAUSE: A Discrepancy Benchmark for Auditing LLMs Legal Reasoning Capabilitiescited by 6
  21. 2025Contradiction Detection in RAG Systems: Evaluating LLMs as Context Validators for Improved Information Consistencycited by 5
  22. 2025To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysiscited by 5
  23. 2025Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large Language Modelscited by 4
  24. 2018A Sheaf Model of Contradictions and Disagreements. Preliminary Report and Discussioncited by 2
  25. 2025Introducing Verification Task of Set Consistency with Set-Consistency Energy Networkscited by 2
  26. 2025Misleading through Inconsistency: A Benchmark for Political Inconsistencies Detectioncited by 1

Built on by

Not cited yet.