Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking

Current automated fact-checking (AFC) approaches typically evaluate evidence either implicitly via the predicted verdicts or through exact matches with predefined closed knowledge sources, such as Wikipedia. However, these methods are limited due to their reliance on evaluation metrics originally designed for other purposes and constraints from closed knowledge sources. In this work, we introduce textbftextcolorskyblueEvtextsuperscript2textcolororangebrownR which combines the strengths of reference-based evaluation and verdict-level proxy scoring. Evtextsuperscript2R jointly assesses how well the evidence aligns with the gold references and how reliably it supports the verdict, addressing the shortcomings of prior methods. We evaluate Evtextsuperscript2R against three types of evidence evaluation approaches: reference-based, proxy-reference, and reference-less baselines. Assessments against human ratings and adversarial tests demonstrate that Evtextsuperscript2R consistently outperforms existing scoring approaches in accuracy and robustness. It achieves stronger correlation with human judgments and greater robustness to adversarial perturbations, establishing it as a reliable metric for evidence evaluation in AFC.footnoteCode is available at hrefhttps://github.com/mubasharaak/fc-evidence-evaluationhttps://github.com/mubasharaak/fc-evidence-evaluation.