mlsanityData to 7 Oct 2026

He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics

arXiv 2107.00807 ↗First posted 2 Jul 2021cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

We investigate how well BERT performs on predicting factuality in several existing English datasets, encompassing various linguistic constructions. Although BERT obtains a strong performance on most datasets, it does so by exploiting common surface patterns that correlate with certain factuality labels, and it fails on instances where pragmatic reasoning is necessary. Contrary to what the high performance suggests, we are still far from having a robust system for factuality prediction.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 11 of its 46 references that are in the map. First posted on arXiv 2 Jul 2021.

Builds on

11 of 46 references.

References · 11

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  4. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  5. 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 76
  6. 2020jiant: A Software Toolkit for Research on General-Purpose Text Understanding Modelscited by 39
  7. 2020ANLIzing the Adversarial Natural Language Inference Datasetcited by 36
  8. 2018Neural models of factualitycited by 20
  9. 2019Harnessing the linguistic signal to predict scalar inferencescited by 17
  10. 2018Lexicosyntactic Inference in Neural Modelscited by 14
  11. 2019Graph based Neural Networks for Event Factuality Prediction using Syntactic and Semantic Structurescited by 9

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 2; 2024: 4; 2025: 1.

12 papers.

  1. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  2. 2022(QA)2: Question Answering with Questionable Assumptionscited by 51
  3. 2024From Single to Multi: How LLMs Hallucinate in Multi-Document Summarizationcited by 17
  4. 2023Finding Common Ground: Annotating and Predicting Common Ground in Spoken Conversationscited by 6
  5. 2022Examining Political Rhetoric with Epistemic Stance Detectioncited by 1
  6. 2024Multimodal Belief Predictioncited by 1
  7. 2024Training LLMs to Recognize Hedges in Spontaneous Narrativescited by 1
  8. 2022Polish Natural Language Inference and Factivity -- an Expert-based Dataset and Benchmarkscited by 0
  9. 2022Crude Oil-related Events Extraction and Processing: A Transfer Learning Approachcited by 0
  10. 2022Measure More, Question More: Experimental Studies on Transformer-based Language Models and Complement Coercioncited by 0
  11. 2024CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Modelcited by 0
  12. 2025Zero-Shot Belief: A Hard Problem for LLMscited by 0