mlsanityData to 14 Oct 2026

Are Large Language Models Good at Detecting Propaganda?

arXiv 2505.13706 ↗First posted 19 May 2025cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Propagandists use rhetorical devices that rely on logical fallacies and emotional appeals to advance their agendas. Recognizing these techniques is key to making informed decisions. Recent advances in Natural Language Processing (NLP) have enabled the development of systems capable of detecting manipulative content. In this study, we look at several Large Language Models and their performance in detecting propaganda techniques in news articles. We compare the performance of these LLMs with transformer-based models. We find that, while GPT-4 demonstrates superior F1 scores (F1=0.16) compared to GPT-3.5 and Claude 3 Opus, it does not outperform a RoBERTa-CRF baseline (F1=0.67). Additionally, we find that all three LLMs outperform a MultiGranularity Network (MGN) baseline in detecting instances of one out of six propaganda techniques (name-calling), with GPT-3.5 and GPT-4 also outperforming the MGN baseline in detecting instances of appeal to fear and flag-waving.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 12 of its 24 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

12 of 24 references.

References · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  5. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  6. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 214
  7. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  8. 2020SemEval-2020 Task 11: Detection of Propaganda Techniques in News Articlescited by 81
  9. 2023Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understandingcited by 40
  10. 2023GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Promptscited by 34
  11. 2020Prta: A System to Support the Analysis of Propaganda Techniques in the Newscited by 24
  12. 2020ApplicaAI at SemEval-2020 Task 11: On RoBERTa-CRF, Span CLS and Whether Self-Training Helps Themcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligencecited by 4
  2. 2025Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuningcited by 1
  3. 2025Fine-grained Narrative Classification in Biased News Articlescited by 1
  4. 2026When Agents Persuade: Rhetoric Generation and Mitigation in LLMscited by 1
  5. 2026Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemascited by 0