# ContraDoc: Understanding Self-Contradictions in Documents with Large Language Models

arXiv: https://arxiv.org/abs/2311.09182 · HTML: https://mlsanity.com/paper/2311.09182/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 55 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [Automated Annotation with Generative AI Requires Validation](https://mlsanity.com/paper/2306.00176/) (2023)
- [DocNLI: A Large-scale Dataset for Document-level Natural Language Inference](https://mlsanity.com/paper/2106.09449/) (2021)
- [I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling](https://mlsanity.com/paper/2012.13391/) (2020)
- [BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation](https://mlsanity.com/paper/2210.07626/) (2022)
- [Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters](https://mlsanity.com/paper/2204.07447/) (2022)
- [WikiContradiction: Detecting Self-Contradiction Articles on Wikipedia](https://mlsanity.com/paper/2111.08543/) (2021)
- [CDConv: A Benchmark for Contradiction Detection in Chinese Conversations](https://mlsanity.com/paper/2210.08511/) (2022)
- [Contradiction Detection for Rumorous Claims](https://mlsanity.com/paper/1611.02588/) (2016)
- [Improving Bot Response Contradiction Detection via Utterance Rewriting](https://mlsanity.com/paper/2207.11862/) (2022)

## Built on by (most cited first, 12 of 12)

- [Taming Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2503.10996/) (2025)
- [Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models](https://mlsanity.com/paper/2505.19509/) (2025)
- [LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documents](https://mlsanity.com/paper/2510.03418/) (2025)
- [HealthContradict: Evaluating Biomedical Knowledge Conflicts in Language Models](https://mlsanity.com/paper/2512.02299/) (2025)
- [Can LLMs Generate User Stories and Assess Their Quality?](https://mlsanity.com/paper/2507.15157/) (2025)
- [Understanding the Dynamics of Demonstration Conflict in In-Context Learning](https://mlsanity.com/paper/2603.04464/) (2026)
- [SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science](https://mlsanity.com/paper/2605.18630/) (2026)
- [Beyond In-Domain Detection: SpikeScore for Cross-Domain Hallucination Detection](https://mlsanity.com/paper/2601.19245/) (2026)
- [The SMeL Test: A simple benchmark for media literacy in language models](https://mlsanity.com/paper/2508.02074/) (2025)
- [Artificial Intelligence Can Match Domain Experts in Evidence Extraction and Critical Appraisal of Microbial Oncogenesis Research Publications](https://mlsanity.com/paper/2608.07250/) (2026)
- [Privacy Washing: Detecting Internal Contradictions in Privacy Policies](https://mlsanity.com/paper/2609.02055/) (2026)
- [ContraVis: Evidence-Grounded Visual Analytics for Contradiction Review in Legal Contracts](https://mlsanity.com/paper/2609.27014/) (2026)

## Abstract (arXiv)

In recent times, large language models (LLMs) have shown impressive performance on various document-level tasks such as document classification, summarization, and question-answering. However, research on understanding their capabilities on the task of self-contradictions in long documents has been very limited. In this work, we introduce ContraDoc, the first human-annotated dataset to study self-contradictions in long documents across multiple domains, varying document lengths, self-contradictions types, and scope. We then analyze the current capabilities of four state-of-the-art open-source and commercially available LLMs: GPT3.5, GPT4, PaLM2, and LLaMAv2 on this dataset. While GPT4 performs the best and can outperform humans on this task, we find that it is still unreliable and struggles with self-contradictions that require more nuance and context. We release the dataset and all the code associated with the experiments (https://github.com/ddhruvkr/CONTRADOC).