# Surprising Efficacy of Fine-Tuned Transformers for Fact-Checking over Larger Language Models

arXiv: https://arxiv.org/abs/2402.12147 · HTML: https://mlsanity.com/paper/2402.12147/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web](https://mlsanity.com/paper/2305.13117/) (2023)
- [Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society](https://mlsanity.com/paper/2005.00033/) (2020)
- [Generating Literal and Implied Subquestions to Fact-check Complex Claims](https://mlsanity.com/paper/2205.06938/) (2022)
- [Claim Detection for Automated Fact-checking: A Survey on Monolingual, Multilingual and Cross-Lingual Research](https://mlsanity.com/paper/2401.11969/) (2024)
- [BRENDA: Browser Extension for Fake News Detection](https://mlsanity.com/paper/2005.13270/) (2020)
- [ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler](https://mlsanity.com/paper/2210.10105/) (2022)

## Built on by (most cited first, 14 of 14)

- [Generative Large Language Models in Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2407.02351/) (2024)
- [Can LLMs Improve Multimodal Fact-Checking by Asking Relevant Questions?](https://mlsanity.com/paper/2410.04616/) (2024)
- [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025)
- [Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs](https://mlsanity.com/paper/2502.08909/) (2025)
- [QuestGen: Effectiveness of Question Generation Methods for Fact-Checking Applications](https://mlsanity.com/paper/2407.21441/) (2024)
- [Efficiency and Effectiveness of LLM-Based Summarization of Evidence in Crowdsourced Fact-Checking](https://mlsanity.com/paper/2501.18265/) (2025)
- [Annotation Tool and Dataset for Fact-Checking Podcasts](https://mlsanity.com/paper/2502.01402/) (2025)
- [Combating Biomedical Misinformation through Multi-modal Claim Detection and Evidence-based Verification](https://mlsanity.com/paper/2509.13888/) (2025)
- [ShortCheck: Checkworthiness Detection of Multilingual Short-Form Videos](https://mlsanity.com/paper/2509.20467/) (2025)
- [NumPert: Numerical Perturbations to Probe Language Models for Veracity Prediction](https://mlsanity.com/paper/2511.09971/) (2025)
- [One LLM to Train Them All: Multi-Task Learning Framework for Fact-Checking](https://mlsanity.com/paper/2601.11293/) (2026)
- [Multilingual Fact-Checking at Scale: Fine-Tuned Compact Models vs LLMs](https://mlsanity.com/paper/2606.08605/) (2026)
- [Calibrating Small Language Models for Claim Check-Worthiness Detection](https://mlsanity.com/paper/2608.30731/) (2026)
- [Towards Robust Numerical Claim Verification](https://mlsanity.com/paper/2610.00689/) (2026)

## Abstract (arXiv)

In this paper, we explore the challenges associated with establishing an end-to-end fact-checking pipeline in a real-world context, covering over 90 languages. Our real-world experimental benchmarks demonstrate that fine-tuning Transformer models specifically for fact-checking tasks, such as claim detection and veracity prediction, provide superior performance over large language models (LLMs) like GPT-4, GPT-3.5-Turbo, and Mistral-7b. However, we illustrate that LLMs excel in generative tasks such as question decomposition for evidence retrieval. Through extensive evaluation, we show the efficacy of fine-tuned models for fact-checking in a multilingual setting and complex claims that include numerical quantities.