# A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments

arXiv: https://arxiv.org/abs/2010.03303 · HTML: https://mlsanity.com/paper/2010.03303/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 10 of 10)

- [From Commit Message Generation to History-Aware Commit Message Completion](https://mlsanity.com/paper/2308.07655/) (2023)
- [LAURA: Enhancing Code Review Generation with Context-Enriched Retrieval-Augmented LLM](https://mlsanity.com/paper/2512.01356/) (2025)
- [Leveraging Reviewer Experience in Code Review Comment Generation](https://mlsanity.com/paper/2409.10959/) (2024)
- [Predicting the First Response Latency of Maintainers and Contributors in Pull Requests](https://mlsanity.com/paper/2311.07786/) (2023)
- [Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents](https://mlsanity.com/paper/2601.20106/) (2026)
- [SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback](https://mlsanity.com/paper/2603.26130/) (2026)
- [On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories](https://mlsanity.com/paper/2604.24450/) (2026)
- [BotHawk: An Approach for Bots Detection in Open Source Software Projects](https://mlsanity.com/paper/2307.13386/) (2023)
- [DevBots can co-design APIs](https://mlsanity.com/paper/2312.05733/) (2023)
- [Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories](https://mlsanity.com/paper/2606.24429/) (2026)

## Abstract (arXiv)

Bots are frequently used in Github repositories to automate repetitive activities that are part of the distributed software development process. They communicate with human actors through comments. While detecting their presence is important for many reasons, no large and representative ground-truth dataset is available, nor are classification models to detect and validate bots on the basis of such a dataset. This paper proposes a ground-truth dataset, based on a manual analysis with high interrater agreement, of pull request and issue comments in 5,000 distinct Github accounts of which 527 have been identified as bots. Using this dataset we propose an automated classification model to detect bots, taking as main features the number of empty and non-empty comments of each account, the number of comment patterns, and the inequality between comments within comment patterns. We obtained a very high weighted average precision, recall and F1-score of 0.98 on a test set containing 40% of the data. We integrated the classification model into an open source command-line tool to allow practitioners to detect which accounts in a given Github repository actually correspond to bots.