# Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence

arXiv: https://arxiv.org/abs/2507.18343 · HTML: https://mlsanity.com/paper/2507.18343/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 40 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Builds on (most cited first, 9 of 9)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019)
- [SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles](https://mlsanity.com/paper/2009.02696/) (2020)
- [Findings of the NLP4IF-2019 Shared Task on Fine-Grained Propaganda Detection](https://mlsanity.com/paper/1910.09982/) (2019)
- [Can GPT-4 Identify Propaganda? Annotation and Detection of Propaganda Spans in News Articles](https://mlsanity.com/paper/2402.17478/) (2024)
- [Large Language Models for Propaganda Span Annotation](https://mlsanity.com/paper/2311.09812/) (2023)
- [Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs](https://mlsanity.com/paper/2502.08909/) (2025)
- [Are Large Language Models Good at Detecting Propaganda?](https://mlsanity.com/paper/2505.13706/) (2025)

## Built on by (most cited first, 4 of 4)

- [Retrieving Climate Change Disinformation by Narrative](https://mlsanity.com/paper/2603.22015/) (2026)
- [Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks](https://mlsanity.com/paper/2602.06526/) (2026)
- [AnnoABSA: A Web-Based Annotation Tool for Aspect-Based Sentiment Analysis with Retrieval-Augmented Suggestions](https://mlsanity.com/paper/2603.01773/) (2026)
- [UkrMeme: Multimodal Explainable Persuasion Detection in Ukrainian Memes](https://mlsanity.com/paper/2610.13783/) (2026)

## Abstract (arXiv)

Propaganda detection on social media remains challenging due to task complexity and limited high-quality labeled data. This paper introduces a novel framework that combines human expertise with Large Language Model (LLM) assistance to improve both annotation consistency and scalability. We propose a hierarchical taxonomy that organizes 14 fine-grained propaganda techniques into three broader categories, conduct a human annotation study on the HQP dataset that reveals low inter-annotator agreement for fine-grained labels, and implement an LLM-assisted pre-annotation pipeline that extracts propagandistic spans, generates concise explanations, and assigns local labels as well as a global label. A secondary human verification study shows significant improvements in both agreement and time-efficiency. Building on this, we fine-tune smaller language models (SLMs) to perform structured annotation. Instead of fine-tuning on human annotations, we train on high-quality LLM-generated data, allowing a large model to produce these annotations and a smaller model to learn to generate them via knowledge distillation. Our work contributes towards the development of scalable and robust propaganda detection systems, supporting the idea of transparent and accountable media ecosystems in line with SDG 16. The code is publicly available at our GitHub repository.