# Are Large Language Models Good at Detecting Propaganda?

arXiv: https://arxiv.org/abs/2505.13706 · HTML: https://mlsanity.com/paper/2505.13706/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 19 May 2025.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](https://mlsanity.com/paper/2303.08559/) (2023)
- [SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articles](https://mlsanity.com/paper/2009.02696/) (2020)
- [Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding](https://mlsanity.com/paper/2304.05368/) (2023)
- [GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts](https://mlsanity.com/paper/2305.12477/) (2023)
- [Prta: A System to Support the Analysis of Propaganda Techniques in the News](https://mlsanity.com/paper/2005.05854/) (2020)
- [ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Them](https://mlsanity.com/paper/2005.07934/) (2020)

## Built on by (most cited first, 5 of 5)

- [Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence](https://mlsanity.com/paper/2507.18343/) (2025)
- [Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuning](https://mlsanity.com/paper/2509.07768/) (2025)
- [Fine-grained Narrative Classification in Biased News Articles](https://mlsanity.com/paper/2512.03582/) (2025)
- [When Agents Persuade: Rhetoric Generation and Mitigation in LLMs](https://mlsanity.com/paper/2603.04636/) (2026)
- [Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas](https://mlsanity.com/paper/2605.13663/) (2026)

## Abstract (arXiv)

Propagandists use rhetorical devices that rely on logical fallacies and emotional appeals to advance their agendas. Recognizing these techniques is key to making informed decisions. Recent advances in Natural Language Processing (NLP) have enabled the development of systems capable of detecting manipulative content. In this study, we look at several Large Language Models and their performance in detecting propaganda techniques in news articles. We compare the performance of these LLMs with transformer-based models. We find that, while GPT-4 demonstrates superior F1 scores (F1=0.16) compared to GPT-3.5 and Claude 3 Opus, it does not outperform a RoBERTa-CRF baseline (F1=0.67). Additionally, we find that all three LLMs outperform a MultiGranularity Network (MGN) baseline in detecting instances of one out of six propaganda techniques (name-calling), with GPT-3.5 and GPT-4 also outperforming the MGN baseline in detecting instances of appeal to fear and flag-waving.