# He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics

arXiv: https://arxiv.org/abs/2107.00807 · HTML: https://mlsanity.com/paper/2107.00807/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 46 references that are in the map.
- First posted on arXiv 2 Jul 2021.

## Builds on (most cited first, 11 of 11)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition](https://mlsanity.com/paper/2004.03066/) (2020)
- [jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models](https://mlsanity.com/paper/2003.02249/) (2020)
- [ANLIzing the Adversarial Natural Language Inference Dataset](https://mlsanity.com/paper/2010.12729/) (2020)
- [Neural models of factuality](https://mlsanity.com/paper/1804.02472/) (2018)
- [Harnessing the linguistic signal to predict scalar inferences](https://mlsanity.com/paper/1910.14254/) (2019)
- [Lexicosyntactic Inference in Neural Models](https://mlsanity.com/paper/1808.06232/) (2018)
- [Graph based Neural Networks for Event Factuality Prediction using Syntactic and Semantic Structures](https://mlsanity.com/paper/1907.03227/) (2019)

## Built on by (most cited first, 12 of 12)

- [Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models](https://mlsanity.com/paper/2302.13439/) (2023)
- [(QA)2: Question Answering with Questionable Assumptions](https://mlsanity.com/paper/2212.10003/) (2022)
- [From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization](https://mlsanity.com/paper/2410.13961/) (2024)
- [Finding Common Ground: Annotating and Predicting Common Ground in Spoken Conversations](https://mlsanity.com/paper/2311.01273/) (2023)
- [Examining Political Rhetoric with Epistemic Stance Detection](https://mlsanity.com/paper/2212.14486/) (2022)
- [Multimodal Belief Prediction](https://mlsanity.com/paper/2406.07466/) (2024)
- [Training LLMs to Recognize Hedges in Spontaneous Narratives](https://mlsanity.com/paper/2408.03319/) (2024)
- [Polish Natural Language Inference and Factivity -- an Expert-based Dataset and Benchmarks](https://mlsanity.com/paper/2201.03521/) (2022)
- [Crude Oil-related Events Extraction and Processing: A Transfer Learning Approach](https://mlsanity.com/paper/2205.00387/) (2022)
- [Measure More, Question More: Experimental Studies on Transformer-based Language Models and Complement Coercion](https://mlsanity.com/paper/2212.10536/) (2022)
- [CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model](https://mlsanity.com/paper/2411.12287/) (2024)
- [Zero-Shot Belief: A Hard Problem for LLMs](https://mlsanity.com/paper/2502.08777/) (2025)

## Abstract (arXiv)

We investigate how well BERT performs on predicting factuality in several existing English datasets, encompassing various linguistic constructions. Although BERT obtains a strong performance on most datasets, it does so by exploiting common surface patterns that correlate with certain factuality labels, and it fails on instances where pragmatic reasoning is necessary. Contrary to what the high performance suggests, we are still far from having a robust system for factuality prediction.