Manufactured Divisiveness: Decomposing the Hostile Content of Seven Social Media Influence Operations
Not on the map. Why?
Abstract (arXiv)
State-backed influence operations are routinely measured as high-prevalence sources of ``hate'' and ``toxicity.'' We argue those rates rest on a measurement error: the detectors behind them are validated to catch a broader definition inclusive of hostility or divisiveness aimed at an out-group, and so over-attribute hate to content better described as partisan or geopolitical invective. Across 25.08M tweets from seven government-attributed campaigns in the Twitter Information Operations archive (8,275 accounts), we separate hate from the other forms of divisiveness. We first validate a two-prompt LLM-based detector, matching human labels at Cohen's kappa=0.82, to identify the broader hostility; we then develop an auditable rule, agreeing with an expert at kappa=0.52, to further classify this content (5,457 posts) into three sub-categories. About 50.1% are identity-based attacks on people, whereas 30.4% are partisan attacks and 19.5% invective against states and their foreign policy. Reporting all of it as hate therefore overstates hate roughly twofold; only 18.7% is both identity-based and dehumanizing or inciting. Six of seven campaigns sort into three regimes that a single ``hate'' rate flattens, namely identity hate (RU-op and IRA, both Russia-attributed), geopolitical invective (both Iran operations), and partisan divisiveness (both Venezuela operations). We call the shared product manufactured divisiveness. The line to separate these constructs itself remains unsettled: on the hardest cases three independent human experts agree only moderately (pairwise kappa=0.37--0.50), and the best of nineteen LLM models tops out at kappa=0.601 against the experts' majority. Our findings can help redefine the study of hate in the context of influence campaigns and broader online discourse.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 56 references that are in the map. First posted on arXiv 16 Jul 2026.
Builds on
28 of 56 references.
References · 28
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
- 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
- 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
- 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
- 2019Measurement and Fairnesscited by 299
- 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 265
- 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
- 2023Can Large Language Models Transform Computational Social Science?cited by 245
- 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
- 2020Directions in Abusive Language Training Data: Garbage In, Garbage Outcited by 139
- 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
- 2017Understanding Abuse: A Typology of Abusive Language Detection Subtaskscited by 115
16 more
- 2018Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaigncited by 30
- 2018Bots increase exposure to negative and inflammatory content in online social systemscited by 26
- 2023From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Modelscited by 26
- 2018Disinformation Warfare: Understanding State-Sponsored Trolls on Twitter and Their Influence on the Webcited by 18
- 2024Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AIcited by 16
- 2021"Short is the Road that Leads from Fear to Hate": Fear Speech in Indian WhatsApp Groupscited by 13
- 2022Retweet-BERT: Political Leaning Detection Using Language Features and Information Diffusion on Social Networkscited by 13
- 2018Who Let The Trolls Out? Towards Understanding State-Sponsored Trollscited by 9
- 2018Measuring bot and human behavioral dynamicscited by 7
- 2023Evidence of Inter-state Coordination amongst State-backed Information Operationscited by 7
- 2024Labeled Datasets for Research on Information Operationscited by 5
- 2024Unraveling the Web of Disinformation: Exploring the Larger Context of State-Sponsored Influence Campaigns on Twittercited by 2
- 2025The Language of Influence: Sentiment, Emotion, and Hate Speech in State Sponsored Influence Operationscited by 2
- 2025Toxicity in State Sponsored Information Operationscited by 2
- 2026RLHF May Not Reflect Genuine Preferencescited by 1
- 2026Cultural Targets, Structural Frames, Binding Morals: A Cross-Lingual Audit of Online Hate in Multicultural Singaporecited by 1
Built on by
Not cited yet.