# MultiVerS: Improving scientific claim verification with weak supervision and full-document context

arXiv: https://arxiv.org/abs/2112.01640 · HTML: https://mlsanity.com/paper/2112.01640/

## Key facts

- Cited by 143 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 2 Dec 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Rationalizing Neural Predictions](https://mlsanity.com/paper/1606.04155/) (2016)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [Decontextualization: Making Sentences Stand-Alone](https://mlsanity.com/paper/2102.05169/) (2021)
- [COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic](https://mlsanity.com/paper/2106.03794/) (2021)
- [An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction](https://mlsanity.com/paper/2005.00652/) (2020)
- [Inferring Which Medical Treatments Work from Reports of Clinical Trials](https://mlsanity.com/paper/1904.01606/) (2019)
- [Towards Few-Shot Fact-Checking via Perplexity](https://mlsanity.com/paper/2103.09535/) (2021)
- [Scientific Claim Verification with VERT5ERINI](https://mlsanity.com/paper/2010.11930/) (2020)
- [Evidence Inference 2.0: More Data, Better Models](https://mlsanity.com/paper/2005.04177/) (2020)
- [A Paragraph-level Multi-task Learning Model for Scientific Fact-Verification](https://mlsanity.com/paper/2012.14500/) (2020)
- [Abstract, Rationale, Stance: A Joint Model for Scientific Claim Verification](https://mlsanity.com/paper/2110.15116/) (2021)
- [Overview and Insights from the SciVer Shared Task on Scientific Claim Verification](https://mlsanity.com/paper/2107.08188/) (2021)
- [Weakly- and Semi-supervised Evidence Extraction](https://mlsanity.com/paper/2011.01459/) (2020)
- [Understanding Clinical Trial Reports: Extracting Medical Entities and Their Relations](https://mlsanity.com/paper/2010.03550/) (2020)

## Built on by (most cited first, 30 of 143)

- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023)
- [SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models](https://mlsanity.com/paper/2311.08370/) (2023)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Handling Bias in Toxic Speech Detection: A Survey](https://mlsanity.com/paper/2202.00126/) (2022)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information](https://mlsanity.com/paper/2301.05036/) (2023)
- [Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Models](https://mlsanity.com/paper/2206.09917/) (2022)
- [SemEval-2023 Task 10: Explainable Detection of Online Sexism](https://mlsanity.com/paper/2303.04222/) (2023)
- [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023)
- [VariErr NLI: Separating Annotation Error from Human Label Variation](https://mlsanity.com/paper/2403.01931/) (2024)
- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels](https://mlsanity.com/paper/2405.05860/) (2024)
- [Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts](https://mlsanity.com/paper/2212.10543/) (2022)
- [LMUnit: Fine-grained Evaluation with Natural Language Unit Tests](https://mlsanity.com/paper/2412.13091/) (2024)
- [Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions](https://mlsanity.com/paper/2502.20897/) (2025)
- [The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation](https://mlsanity.com/paper/2405.01299/) (2024)
- [Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directions](https://mlsanity.com/paper/2403.01222/) (2024)
- [IssueBench: Millions of Realistic Prompts for Measuring Issue Bias in LLM Writing Assistance](https://mlsanity.com/paper/2502.08395/) (2025)
- [Hate Personified: Investigating the role of LLMs in content moderation](https://mlsanity.com/paper/2410.02657/) (2024)
- [Beyond Denouncing Hate: Strategies for Countering Implied Biases and Stereotypes in Language](https://mlsanity.com/paper/2311.00161/) (2023)
- [Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement](https://mlsanity.com/paper/2301.10684/) (2023)
- [What's the Meaning of Superhuman Performance in Today's NLU?](https://mlsanity.com/paper/2305.08414/) (2023)
- [NBIAS: A Natural Language Processing Framework for Bias Identification in Text](https://mlsanity.com/paper/2308.01681/) (2023)
- [Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas](https://mlsanity.com/paper/2406.14462/) (2024)
- [How Far Can We Extract Diverse Perspectives from Large Language Models?](https://mlsanity.com/paper/2311.09799/) (2023)
- [Conformalized Credal Set Predictors](https://mlsanity.com/paper/2402.10723/) (2024)
- [Personas with Attitudes: Controlling LLMs for Diverse Data Annotation](https://mlsanity.com/paper/2410.11745/) (2024)
- [The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youth](https://mlsanity.com/paper/2402.11886/) (2024)

## Abstract (arXiv)

The scientific claim verification task requires an NLP system to label scientific documents which Support or Refute an input claim, and to select evidentiary sentences (or rationales) justifying each predicted label. In this work, we present MultiVerS, which predicts a fact-checking label and identifies rationales in a multitask fashion based on a shared encoding of the claim and full document context. This approach accomplishes two key modeling goals. First, it ensures that all relevant contextual information is incorporated into each labeling decision. Second, it enables the model to learn from instances annotated with a document-level fact-checking label, but lacking sentence-level rationales. This allows MultiVerS to perform weakly-supervised domain adaptation by training on scientific documents labeled using high-precision heuristics. Our approach outperforms two competitive baselines on three scientific claim verification datasets, with particularly strong performance in zero / few-shot domain adaptation experiments. Our code and data are available at https://github.com/dwadden/multivers.