# Fact or Fiction? Can LLMs be Reliable Annotators for Political Truths?

arXiv: https://arxiv.org/abs/2411.05775 · HTML: https://mlsanity.com/paper/2411.05775/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 8 Nov 2024.

## Builds on (most cited first, 13 of 13)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Large Language Models for Data Annotation and Synthesis: A Survey](https://mlsanity.com/paper/2402.13446/) (2024)
- [MEGAnno+: A Human-LLM Collaborative Annotation System](https://mlsanity.com/paper/2402.18050/) (2024)
- [Leveraging Large Language Models for NLG Evaluation: Advances and Challenges](https://mlsanity.com/paper/2401.07103/) (2024)
- [Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning](https://mlsanity.com/paper/2307.02179/) (2023)
- [Multi-News+: Cost-efficient Dataset Cleansing via LLM-based Data Annotation](https://mlsanity.com/paper/2404.09682/) (2024)
- [Investigating Annotator Bias in Large Language Models for Hate Speech Detection](https://mlsanity.com/paper/2406.11109/) (2024)

## Built on by (most cited first, 3 of 3)

- [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025)
- [MPCG: Multi-Round Persona-Conditioned Generation for Modeling the Evolution of Misinformation with LLMs](https://mlsanity.com/paper/2509.16564/) (2025)
- [Effect of Selection Format on LLM Performance](https://mlsanity.com/paper/2503.06926/) (2025)

## Abstract (arXiv)

Political misinformation poses significant challenges to democratic processes, shaping public opinion and trust in media. Manual fact-checking methods face issues of scalability and annotator bias, while machine learning models require large, costly labelled datasets. This study investigates the use of state-of-the-art large language models (LLMs) as reliable annotators for detecting political factuality in news articles. Using open-source LLMs, we create a politically diverse dataset, labelled for bias through LLM-generated annotations. These annotations are validated by human experts and further evaluated by LLM-based judges to assess the accuracy and reliability of the annotations. Our approach offers a scalable and robust alternative to traditional fact-checking, enhancing transparency and public trust in media.