# Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking

arXiv: https://arxiv.org/abs/2411.05375 · HTML: https://mlsanity.com/paper/2411.05375/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 63 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://mlsanity.com/paper/2103.08541/) (2021)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://mlsanity.com/paper/2011.03088/) (2020)
- [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://mlsanity.com/paper/1909.03242/) (2019)
- [A Survey of Evaluation Metrics Used for NLG Systems](https://mlsanity.com/paper/2008.12009/) (2020)
- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [Combining Fact Extraction and Verification with Neural Semantic Matching Networks](https://mlsanity.com/paper/1811.07039/) (2018)
- [Complex Claim Verification with Evidence Retrieved in the Wild](https://mlsanity.com/paper/2305.11859/) (2023)
- [The Fact Extraction and VERification (FEVER) Shared Task](https://mlsanity.com/paper/1811.10971/) (2018)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)

## Built on by (most cited first, 18 of 18)

- [AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web](https://mlsanity.com/paper/2505.17978/) (2025)
- [The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers](https://mlsanity.com/paper/2506.20844/) (2025)
- [Team HUMANE at AVeriTeC 2025: HerO 2 for Efficient Fact Verification](https://mlsanity.com/paper/2507.11004/) (2025)
- [MR-Align: Meta-Reasoning Informed Factuality Alignment for Large Reasoning Models](https://mlsanity.com/paper/2510.24794/) (2025)
- [AIC CTU@FEVER 8: On-premise fact checking through long context RAG](https://mlsanity.com/paper/2508.04390/) (2025)
- [TSVer: A Benchmark for Fact Verification Against Time-Series Evidence](https://mlsanity.com/paper/2511.01101/) (2025)
- [The Automatic Verification of Image-Text Claims (AVerImaTeC) Shared Task](https://mlsanity.com/paper/2602.11221/) (2026)
- [FActBench: A Benchmark for Fine-grained Automatic Evaluation of LLM-Generated Text in the Medical Domain](https://mlsanity.com/paper/2509.02198/) (2025)
- [Multimodal Claim Extraction for Fact-Checking](https://mlsanity.com/paper/2604.16311/) (2026)
- [VILLAIN at AVerImaTeC: Verifying Image-Text Claims via Multi-Agent Collaboration](https://mlsanity.com/paper/2602.04587/) (2026)
- [AIC CTU@AVerImaTeC: dual-retriever RAG for image-text fact checking](https://mlsanity.com/paper/2602.15190/) (2026)
- [ClimateCheck 2026: Scientific Fact-Checking and Disinformation Narrative Classification of Climate-related Claims](https://mlsanity.com/paper/2603.26449/) (2026)
- [CAAFC: Chronological Actionable Automated Fact-Checker for misinformation / non-factual hallucination detection and correction](https://mlsanity.com/paper/2605.12436/) (2026)
- [I Know What You Meme, Even If it Emerged Today: Understanding Evolving Memes through Open-World Knowledge Acquisition](https://mlsanity.com/paper/2606.05316/) (2026)
- [The Warrant Gap: Claim-Conditioned Re-scoring for Fact-Checking](https://mlsanity.com/paper/2606.24627/) (2026)
- [ElementCheck: Complexity-Aware Long-Form Text Factuality Evaluation via Sentence Elements](https://mlsanity.com/paper/2608.26118/) (2026)
- [PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning](https://mlsanity.com/paper/2609.04598/) (2026)
- [Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation](https://mlsanity.com/paper/2609.08943/) (2026)

## Abstract (arXiv)

Current automated fact-checking (AFC) approaches typically evaluate evidence either implicitly via the predicted verdicts or through exact matches with predefined closed knowledge sources, such as Wikipedia. However, these methods are limited due to their reliance on evaluation metrics originally designed for other purposes and constraints from closed knowledge sources. In this work, we introduce textbftextcolorskyblueEvtextsuperscript2textcolororangebrownR which combines the strengths of reference-based evaluation and verdict-level proxy scoring. Evtextsuperscript2R jointly assesses how well the evidence aligns with the gold references and how reliably it supports the verdict, addressing the shortcomings of prior methods. We evaluate Evtextsuperscript2R against three types of evidence evaluation approaches: reference-based, proxy-reference, and reference-less baselines. Assessments against human ratings and adversarial tests demonstrate that Evtextsuperscript2R consistently outperforms existing scoring approaches in accuracy and robustness. It achieves stronger correlation with human judgments and greater robustness to adversarial perturbations, establishing it as a reliable metric for evidence evaluation in AFC.footnoteCode is available at hrefhttps://github.com/mubasharaak/fc-evidence-evaluationhttps://github.com/mubasharaak/fc-evidence-evaluation.