# Simplistic Collection and Labeling Practices Limit the Utility of Benchmark Datasets for Twitter Bot Detection

arXiv: https://arxiv.org/abs/2301.07015 · HTML: https://mlsanity.com/paper/2301.07015/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 78 references that are in the map.
- First posted on arXiv 17 Jan 2023.

## Builds on (most cited first, 28 of 28)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [The spread of low-credibility content by social bots](https://mlsanity.com/paper/1707.07592/) (2017)
- [Scalable and Generalizable Social Bot Detection through Data Selection](https://mlsanity.com/paper/1911.09179/) (2019)
- [TwiBot-20: A Comprehensive Twitter Bot Detection Benchmark](https://mlsanity.com/paper/2106.13088/) (2021)
- [The paradigm-shift of social spambots: Evidence, theories, and tools for the arms race](https://mlsanity.com/paper/1701.03017/) (2017)
- [A Decade of Social Bot Detection](https://mlsanity.com/paper/2007.03604/) (2020)
- [Heterogeneity-aware Twitter Bot Detection with Relational Graph Transformers](https://mlsanity.com/paper/2109.02927/) (2021)
- [BotRGCN: Twitter Bot Detection with Relational Graph Convolutional Networks](https://mlsanity.com/paper/2106.13092/) (2021)
- [Deep Neural Networks for Bot Detection](https://mlsanity.com/paper/1802.04289/) (2018)
- [Fame for sale: efficient detection of fake Twitter followers](https://mlsanity.com/paper/1509.04098/) (2015)
- [Detection of Novel Social Bots by Ensembles of Specialized Classifiers](https://mlsanity.com/paper/2006.06867/) (2020)
- [Analyzing the Digital Traces of Political Manipulation: The 2016 Russian Interference Twitter Campaign](https://mlsanity.com/paper/1802.04291/) (2018)
- [RTbust: Exploiting Temporal Patterns for Botnet Detection on Twitter](https://mlsanity.com/paper/1902.04506/) (2019)
- [Disinformation and Social Bot Operations in the Run Up to the 2017 French Presidential Election](https://mlsanity.com/paper/1707.00086/) (2017)
- [Bots increase exposure to negative and inflammatory content in online social systems](https://mlsanity.com/paper/1802.07292/) (2018)
- [Botometer 101: Social bot practicum for computational social scientists](https://mlsanity.com/paper/2201.01608/) (2022)
- [Arming the public with artificial intelligence to counter social bots](https://mlsanity.com/paper/1901.00912/) (2019)
- [SATAR: A Self-supervised Approach to Twitter Account Representation Learning and its Application in Bot Detection](https://mlsanity.com/paper/2106.13089/) (2021)
- [DNA-inspired online behavioral modeling and its application to spambot detection](https://mlsanity.com/paper/1602.00110/) (2016)
- [Charting the Landscape of Online Cryptocurrency Manipulation](https://mlsanity.com/paper/2001.10289/) (2020)
- [LOBO -- Evaluation of Generalization Deficiencies in Twitter Bot Classifiers](https://mlsanity.com/paper/1809.09684/) (2018)
- [Anatomy of an online misinformation network](https://mlsanity.com/paper/1801.06122/) (2018)
- [Characterizing Retweet Bots: The Case of Black Market Accounts](https://mlsanity.com/paper/2112.02366/) (2021)
- [Investigating Italian disinformation spreading on Twitter in the context of 2019 European elections](https://mlsanity.com/paper/1907.08170/) (2019)
- [Investigating the Validity of Botometer-based Social Bot Studies](https://mlsanity.com/paper/2207.11474/) (2022)

## Built on by (most cited first, 13 of 13)

- [Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI](https://mlsanity.com/paper/2308.07213/) (2023)
- [Demystifying Misconceptions in Social Bots Research](https://mlsanity.com/paper/2303.17251/) (2023)
- [Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments](https://mlsanity.com/paper/2505.22137/) (2025)
- [Boosting Bot Detection via Heterophily-Aware Representation Learning and Prototype-Guided Cluster Discovery](https://mlsanity.com/paper/2506.00989/) (2025)
- [How Good Are SOTA Fake News Detectors](https://mlsanity.com/paper/2308.02727/) (2023)
- [HausaNLP: Current Status, Challenges and Future Directions for Hausa Natural Language Processing](https://mlsanity.com/paper/2505.14311/) (2025)
- [BotHawk: An Approach for Bots Detection in Open Source Software Projects](https://mlsanity.com/paper/2307.13386/) (2023)
- [Certainly Bot Or Not? Trustworthy Social Bot Detection via Robust Multi-Modal Neural Processes](https://mlsanity.com/paper/2503.09626/) (2025)
- [SPADE: Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detection](https://mlsanity.com/paper/2503.15044/) (2025)
- [BounTCHA: A CAPTCHA Utilizing Boundary Identification in Guided Generative AI-extended Videos](https://mlsanity.com/paper/2501.18565/) (2025)
- [Bot Meets Shortcut: How Can LLMs Aid in Handling Unknown Invariance OOD Scenarios?](https://mlsanity.com/paper/2511.08455/) (2025)
- [Multi-label versus multi-class classification of blood cells and their aggregates in microfluidic channels](https://mlsanity.com/paper/2609.07410/) (2026)
- [FISSION: Label Augmentation for Bot Detection](https://mlsanity.com/paper/2609.26279/) (2026)

## Abstract (arXiv)

Accurate bot detection is necessary for the safety and integrity of online platforms. It is also crucial for research on the influence of bots in elections, the spread of misinformation, and financial market manipulation. Platforms deploy infrastructure to flag or remove automated accounts, but their tools and data are not publicly available. Thus, the public must rely on third-party bot detection. These tools employ machine learning and often achieve near perfect performance for classification on existing datasets, suggesting bot detection is accurate, reliable and fit for use in downstream applications. We provide evidence that this is not the case and show that high performance is attributable to limitations in dataset collection and labeling rather than sophistication of the tools. Specifically, we show that simple decision rules -- shallow decision trees trained on a small number of features -- achieve near-state-of-the-art performance on most available datasets and that bot detection datasets, even when combined together, do not generalize well to out-of-sample datasets. Our findings reveal that predictions are highly dependent on each dataset's collection and labeling procedures rather than fundamental differences between bots and humans. These results have important implications for both transparency in sampling and labeling procedures and potential biases in research using existing bot detection tools for pre-processing.