What Changes When Fact-Verification Scores Improve? Evidence and Answer Accounting Across Trained Verifiers and LLMs
Not on the map. Why?
Abstract (arXiv)
A joint fact-verification score assesses answers and submitted evidence together. When the score improves, how much of the gain remains if the answers are held fixed? On FEVEROUS, strict score is the percentage of claims with a correct answer and a complete annotated evidence group in the submitted evidence. Across four trained DeBERTa checkpoints and 7,890 claims, replacing DCUF evidence with UnifEE evidence raises strict score by 9.61 percentage points, compared with 1.96 percentage points in answer accuracy. The paired 95% interval for the strict-score gain is [8.77, 10.43], conditional on these checkpoints. Replacing only the evidence passed to the scorer accounts for 7.92 or 9.08 percentage points when we retain the answers generated from DCUF or UnifEE evidence, respectively. To examine how this evidence gain depends on evaluation choices, we generate 470,400 responses from two 8B LLMs on FEVER, FEVEROUS, and SciFact under two answer formats and two context budgets. Increasing context from 256 to 2,048 tokens raises the fixed-answer evidence gain on FEVEROUS by 3.84 and 3.10 percentage points for Qwen and Llama, respectively. The effects fall short of the prespecified cross-dataset criterion, while some intervals extend beyond the two-point small-effect bound. Post-hoc analyses quantify changes in answers and submitted evidence, and show when aggregate accuracy and evidence-coverage rates miss the claim-level pattern. The four answer-evidence score combinations reveal changes that endpoint and aggregate metrics leave unresolved.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 22 Sep 2026.
Builds on
16 of 23 references.
References · 16
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
- 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
- 2023Enabling Large Language Models to Generate Text with Citationscited by 584
- 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
- 2020Fact or Fiction: Verifying Scientific Claimscited by 546
- 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
- 2019Towards Debiasing Fact Verification Modelscited by 143
4 more
- 2020With Little Power Comes Great Responsibilitycited by 138
- 2025Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMscited by 23
- 2024Attribute or Abstain: Large Language Models as Long Document Assistantscited by 10
- 2026Context Shapes LLMs Retrieval-Augmented Fact-Checking Effectivenesscited by 1
Built on by
Not cited yet.