mlsanityData to 7 Oct 2026

Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence

arXiv 2103.08541 ↗First posted 15 Mar 2021cs.CL cs.IR cs.LGcited by 222

Not on the map. Why?

Abstract (arXiv)

Typical fact verification models use retrieved written evidence to verify claims. Evidence sources, however, often change over time as more information is gathered and revised. In order to adapt, models must be sensitive to subtle differences in supporting evidence. We present VitaminC, a benchmark infused with challenging cases that require fact verification models to discern and adjust to slight factual changes. We collect over 100,000 Wikipedia revisions that modify an underlying fact, and leverage these revisions, together with additional synthetically constructed ones, to create a total of over 400,000 claim-evidence pairs. Unlike previous resources, the examples in VitaminC are contrastive, i.e., they contain evidence pairs that are nearly identical in language and content, with the exception that one supports a given claim while the other does not. We show that training using this design increases robustness -- improving accuracy by 10% on adversarial fact verification and 6% on adversarial natural language inference (NLI). Moreover, the structure of VitaminC leads us to define additional tasks for fact-checking resources: tagging relevant words in the evidence for verifying the claim, identifying factual revisions, and providing automatic edits via factually consistent text generation.

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). Builds on 55 of its 82 references that are in the map. First posted on arXiv 15 Mar 2021.

Builds on

55 of 82 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  12. 2016Enriching Word Vectors with Subword Informationcited by 1,578
38 more
  1. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  5. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  6. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  7. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  8. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  9. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  10. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  11. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  12. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 523
  13. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  14. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  15. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  16. 2017"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detectioncited by 361
  17. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  18. 2016Rationalizing Neural Predictionscited by 281
  19. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  20. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 199
  21. 2019MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claimscited by 169
  22. 2020Adversarial Filters of Dataset Biasescited by 159
  23. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  24. 2019Towards Debiasing Fact Verification Modelscited by 143
  25. 2020Generating Fact Checking Explanationscited by 132
  26. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 127
  27. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  28. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  29. 2020Explaining NLP Models via Minimal Contrastive Editing (MiCE)cited by 105
  30. 2018DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learningcited by 100
  31. 2018Automated Fact Checking: Task formulations, methods and future directionscited by 96
  32. 2020Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performancecited by 85
  33. 2019The Limitations of Stylometry for Detecting Machine-Generated Fake Newscited by 63
  34. 2019Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generationcited by 62
  35. 2018UKP-Athene: Multi-Sentence Textual Entailment for Claim Verificationcited by 60
  36. 2018Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailmentcited by 58
  37. 2020Generating Fact Checking Briefscited by 50
  38. 2019Generating Token-Level Explanations for Natural Language Inferencecited by 28

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 27; 2023: 55; 2024: 65; 2025: 28; 2026: 28.

222 papers.

Sort:

By area:

  1. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  2. 2022Memory-Based Model Editing at Scalecited by 443
  3. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  4. 2021A Survey on Automated Fact-Checkingcited by 376
  5. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  6. 2022UL2: Unifying Language Learning Paradigmscited by 292
  7. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  8. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  9. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  10. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  11. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  12. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
38 more
  1. 2023Conformal Language Modelingcited by 163
  2. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  3. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  4. 2023Fact-Checking Complex Claims with Program-Guided Reasoningcited by 146
  5. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  6. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  7. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  8. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  9. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  10. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  11. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  12. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  13. 2022SciFact-Open: Towards open-domain scientific claim verificationcited by 94
  14. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  15. 2023Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkerscited by 86
  16. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  17. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  18. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  19. 2021Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resourcescited by 71
  20. 2023Automatic Evaluation of Attribution by Large Language Modelscited by 70
  21. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  22. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
  23. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  24. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 63
  25. 2021DialFact: A Benchmark for Fact-Checking in Dialoguecited by 62
  26. 2023FactKG: Fact Verification via Reasoning on Knowledge Graphscited by 57
  27. 2021Automated Fact-Checking: A Surveycited by 55
  28. 2021ProoFVer: Natural Logic Theorem Proving for Fact Verificationcited by 52
  29. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  30. 2023SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tablescited by 51
  31. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  32. 2024Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Modelscited by 51
  33. 2023Do Large Language Models Know about Facts?cited by 49
  34. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  35. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  36. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 45
  37. 2022Synthetic Disinformation Attacks on Automated Fact Verification Systemscited by 44
  38. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44

50 most cited of 222 All 222 (JSON)