# GPT Assisted Annotation of Rhetorical and Linguistic Features for Interpretable Propaganda Technique Detection in News Text arXiv: https://arxiv.org/abs/2407.11827 ยท HTML: https://mlsanity.com/paper/2407.11827/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 20 of its 33 references that are in the map. - First posted on arXiv 16 Jul 2024. ## Builds on (most cited first, 20 of 20) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022) - [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023) - [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021) - [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021) - [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022) - [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021) - [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023) - [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023) - [Fine-Grained Analysis of Propaganda in News Articles](https://mlsanity.com/paper/1910.02517/) (2019) - [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023) - [LLMaAA: Making Large Language Models as Active Annotators](https://mlsanity.com/paper/2310.19596/) (2023) - [Automated Annotation with Generative AI Requires Validation](https://mlsanity.com/paper/2306.00176/) (2023) - [Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark](https://mlsanity.com/paper/2304.11085/) (2023) - [ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification](https://mlsanity.com/paper/2303.03953/) (2023) - [Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts](https://mlsanity.com/paper/2305.04417/) (2023) - [Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost](https://mlsanity.com/paper/2306.15766/) (2023) - [Interpretable Propaganda Detection in News Articles](https://mlsanity.com/paper/2108.12802/) (2021) - [Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmenting](https://mlsanity.com/paper/2309.08108/) (2023) - [Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweets](https://mlsanity.com/paper/2309.06503/) (2023) ## Built on by (most cited first, 4 of 4) - [Polite But Boring? Trade-offs Between Engagement and Psychological Reactance to Chatbot Feedback Styles](https://mlsanity.com/paper/2601.20683/) (2026) - [The Tag is the Signal: URL-Agnostic Credibility Scoring for Messages on Telegram](https://mlsanity.com/paper/2601.13294/) (2026) - [Wisdom of the LLM Crowd: A Large Scale Benchmark of Multi-Label U.S. Election-Related Harmful Social Media Content](https://mlsanity.com/paper/2602.11962/) (2026) - [Challenges in annotations by humans and LLMs: A case study of evaluative language](https://mlsanity.com/paper/2607.28119/) (2026) ## Abstract (arXiv) While the use of machine learning for the detection of propaganda techniques in text has garnered considerable attention, most approaches focus on "black-box" solutions with opaque inner workings. Interpretable approaches provide a solution, however, they depend on careful feature engineering and costly expert annotated data. Additionally, language features specific to propagandistic text are generally the focus of rhetoricians or linguists, and there is no data set labeled with such features suitable for machine learning. This study codifies 22 rhetorical and linguistic features identified in literature related to the language of persuasion for the purpose of annotating an existing data set labeled with propaganda techniques. To help human experts annotate natural language sentences with these features, RhetAnn, a web application, was specifically designed to minimize an otherwise considerable mental effort. Finally, a small set of annotated data was used to fine-tune GPT-3.5, a generative large language model (LLM), to annotate the remaining data while optimizing for financial cost and classification accuracy. This study demonstrates how combining a small number of human annotated examples with GPT can be an effective strategy for scaling the annotation process at a fraction of the cost of traditional annotation relying solely on human experts. The results are on par with the best performing model at the time of writing, namely GPT-4, at 10x less the cost. Our contribution is a set of features, their properties, definitions, and examples in a machine-readable format, along with the code for RhetAnn and the GPT prompts and fine-tuning procedures for advancing state-of-the-art interpretable propaganda technique detection.