# Detection of Conspiracy Theories Beyond Keyword Bias in German-Language Telegram Using Large Language Models

arXiv: https://arxiv.org/abs/2404.17985 · HTML: https://mlsanity.com/paper/2404.17985/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 80 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [Latent Hatred: A Benchmark for Understanding Implicit Hate Speech](https://mlsanity.com/paper/2109.05322/) (2021)
- [Active Example Selection for In-Context Learning](https://mlsanity.com/paper/2211.04486/) (2022)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](https://mlsanity.com/paper/2005.07503/) (2020)
- [Detecting Hate Speech with GPT-3](https://mlsanity.com/paper/2103.12407/) (2021)
- ["HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media](https://mlsanity.com/paper/2304.10619/) (2023)
- [Toxicity Detection with Generative Prompt-based Inference](https://mlsanity.com/paper/2205.12390/) (2022)
- [Open, Closed, or Small Language Models for Text Classification?](https://mlsanity.com/paper/2308.10092/) (2023)
- [Characterizing COVID-19 Misinformation Communities Using a Novel Twitter Dataset](https://mlsanity.com/paper/2008.00791/) (2020)
- [From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models](https://mlsanity.com/paper/2305.17174/) (2023)
- [On the Globalization of the QAnon Conspiracy Theory Through Telegram](https://mlsanity.com/paper/2105.13020/) (2021)
- [Reliability Check: An Analysis of GPT-3's Response to Sensitive Topics and Prompt Wording](https://mlsanity.com/paper/2306.06199/) (2023)
- [Identifying Conspiracy Theories News based on Event Relation Graph](https://mlsanity.com/paper/2310.18545/) (2023)
- [Codes, Patterns and Shapes of Contemporary Online Antisemitism and Conspiracy Narratives -- an Annotation Guide and Labeled German-Language Dataset in the Context of COVID-19](https://mlsanity.com/paper/2210.07934/) (2022)
- [Human-in-the-Loop Hate Speech Classification in a Multilingual Context](https://mlsanity.com/paper/2212.02108/) (2022)
- [Domain Adaptation of Transformer-Based Models using Unlabeled Data for Relevance and Polarity Classification of German Customer Feedback](https://mlsanity.com/paper/2212.05764/) (2022)

## Built on by (most cited first, 4 of 4)

- [Debunking with Dialogue? Exploring AI-Generated Counterspeech to Challenge Conspiracy Theories](https://mlsanity.com/paper/2504.16604/) (2025)
- [ConspirED: A Dataset for Cognitive Traits of Conspiracy Theories and Large Language Model Safety](https://mlsanity.com/paper/2508.20468/) (2025)
- [AILS-NTUA at SemEval-2026 Task 10: Agentic LLMs for Psycholinguistic Marker Extraction and Conspiracy Endorsement Detection](https://mlsanity.com/paper/2603.04921/) (2026)
- [Who's Behind It? Annotating and Extracting Conspiratorial Actors from German Telegram Posts](https://mlsanity.com/paper/2607.04962/) (2026)

## Abstract (arXiv)

The automated detection of conspiracy theories online typically relies on supervised learning. However, creating respective training data requires expertise, time and mental resilience, given the often harmful content. Moreover, available datasets are predominantly in English and often keyword-based, introducing a token-level bias into the models. Our work addresses the task of detecting conspiracy theories in German Telegram messages. We compare the performance of supervised fine-tuning approaches using BERT-like models with prompt-based approaches using Llama2, GPT-3.5, and GPT-4 which require little or no additional training data. We use a dataset of ~!! 4,000 messages collected during the COVID-19 pandemic, without the use of keyword filters. Our findings demonstrate that both approaches can be leveraged effectively: For supervised fine-tuning, we report an F1 score of ~!! 0.8 for the positive class, making our model comparable to recent models trained on keyword-focused English corpora. We demonstrate our model's adaptability to intra-domain temporal shifts, achieving F1 scores of ~!! 0.7. Among prompting variants, the best model is GPT-4, achieving an F1 score of ~!! 0.8 for the positive class in a zero-shot setting and equipped with a custom conspiracy theory definition.