# Large Language Models for Propaganda Span Annotation

arXiv: https://arxiv.org/abs/2311.09812 · HTML: https://mlsanity.com/paper/2311.09812/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://mlsanity.com/paper/2007.13184/) (2020)
- [A Survey on Computational Propaganda Detection](https://mlsanity.com/paper/2007.08024/) (2020)
- [Detecting and Understanding Harmful Memes: A Survey](https://mlsanity.com/paper/2205.04274/) (2022)
- [SemEval-2021 Task 6: Detection of Persuasion Techniques in Texts and Images](https://mlsanity.com/paper/2105.09284/) (2021)
- [Detecting Propaganda Techniques in Memes](https://mlsanity.com/paper/2109.08013/) (2021)
- [AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding](https://mlsanity.com/paper/2012.15516/) (2020)
- [Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies](https://mlsanity.com/paper/2309.13063/) (2023)
- [Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost](https://mlsanity.com/paper/2306.15766/) (2023)
- [Open-Source LLMs for Text Annotation: A Practical Guide for Model Setting and Fine-Tuning](https://mlsanity.com/paper/2307.02179/) (2023)
- [Multitask Instruction-based Prompting for Fallacy Recognition](https://mlsanity.com/paper/2301.09992/) (2023)
- [LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking](https://mlsanity.com/paper/2308.04945/) (2023)
- [ArAIEval Shared Task: Persuasion Techniques and Disinformation Detection in Arabic Text](https://mlsanity.com/paper/2311.03179/) (2023)

## Built on by (most cited first, 19 of 19)

- [Towards Interpretable Hate Speech Detection using Large Language Model-extracted Rationales](https://mlsanity.com/paper/2403.12403/) (2024)
- [FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers](https://mlsanity.com/paper/2511.21843/) (2025)
- [ArAIEval Shared Task: Propagandistic Techniques Detection in Unimodal and Multimodal Arabic Content](https://mlsanity.com/paper/2407.04247/) (2024)
- [Propaganda to Hate: A Multimodal Analysis of Arabic Memes with Multi-Agent LLMs](https://mlsanity.com/paper/2409.07246/) (2024)
- [LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content](https://mlsanity.com/paper/2410.15308/) (2024)
- [Can Third-parties Read Our Emotions?](https://mlsanity.com/paper/2504.18673/) (2025)
- [Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence](https://mlsanity.com/paper/2507.18343/) (2025)
- [Nullpointer at ArAIEval Shared Task: Arabic Propagandist Technique Detection with Token-to-Word Mapping in Sequence Tagging](https://mlsanity.com/paper/2407.01360/) (2024)
- [Crowdsourcing Lexical Diversity](https://mlsanity.com/paper/2410.23133/) (2024)
- [ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety](https://mlsanity.com/paper/2508.20468/) (2025)
- [MemeMind at ArAIEval Shared Task: Spotting Persuasive Spans in Arabic Text with Persuasion Techniques Identification](https://mlsanity.com/paper/2408.04540/) (2024)
- [The Tag is the Signal: URL-Agnostic Credibility Scoring for Messages on Telegram](https://mlsanity.com/paper/2601.13294/) (2026)
- [Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis](https://mlsanity.com/paper/2601.16800/) (2026)
- [CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents](https://mlsanity.com/paper/2603.03884/) (2026)
- [CritiSense: Critical Digital Literacy and Resilience Against Misinformation](https://mlsanity.com/paper/2603.16672/) (2026)
- [Semantically Enriching Investor Micro-blogs for Opinion-Aware Emotion Analysis: A Practical Approach](https://mlsanity.com/paper/2605.03092/) (2026)
- [Fine-tuning with Hierarchical Prompting for Robust Propaganda Classification Across Annotation Schemas](https://mlsanity.com/paper/2605.13663/) (2026)
- [Cross-National Information Attacks: A Two-Decade Analysis of Troll Behavior in Korea](https://mlsanity.com/paper/2606.22785/) (2026)
- [LAAF: A Layered Accountability Architecture Framework for LLM Applications](https://mlsanity.com/paper/2608.27102/) (2026)

## Abstract (arXiv)

The use of propagandistic techniques in online content has increased in recent years aiming to manipulate online audiences. Fine-grained propaganda detection and extraction of textual spans where propaganda techniques are used, are essential for more informed content consumption. Automatic systems targeting the task over lower resourced languages are limited, usually obstructed by lack of large scale training datasets. Our study investigates whether Large Language Models (LLMs), such as GPT-4, can effectively extract propagandistic spans. We further study the potential of employing the model to collect more cost-effective annotations. Finally, we examine the effectiveness of labels provided by GPT-4 in training smaller language models for the task. The experiments are performed over a large-scale in-house manually annotated dataset. The results suggest that providing more annotation context to GPT-4 within prompts improves its performance compared to human annotators. Moreover, when serving as an expert annotator (consolidator), the model provides labels that have higher agreement with expert annotators, and lead to specialized models that achieve state-of-the-art over an unseen Arabic testing set. Finally, our work is the first to show the potential of utilizing LLMs to develop annotated datasets for propagandistic spans detection task prompting it with annotations from human annotators with limited expertise. All scripts and annotations will be shared with the community.