# Bot or Human? Detecting ChatGPT Imposters with A Single Question

arXiv: https://arxiv.org/abs/2305.06424 · HTML: https://mlsanity.com/paper/2305.06424/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 10 May 2023.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [Measuring and Improving Consistency in Pretrained Language Models](https://mlsanity.com/paper/2102.01017/) (2021)
- [Graph-Bert: Only Attention is Needed for Learning Graph Representations](https://mlsanity.com/paper/2001.05140/) (2020)
- [Real or Fake? Learning to Discriminate Machine from Human Generated Text](https://mlsanity.com/paper/1906.03351/) (2019)
- [Protecting Language Generation Models via Invisible Watermarking](https://mlsanity.com/paper/2302.03162/) (2023)
- [ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text](https://mlsanity.com/paper/2301.13852/) (2023)
- [Limitations of Language Models in Arithmetic and Symbolic Induction](https://mlsanity.com/paper/2208.05051/) (2022)
- [Visually-Augmented Language Modeling](https://mlsanity.com/paper/2205.10178/) (2022)
- [Task-Oriented Dialogue System as Natural Language Generation](https://mlsanity.com/paper/2108.13679/) (2021)
- [Gotta CAPTCHA 'Em All: A Survey of Twenty years of the Human-or-Computer Dilemma](https://mlsanity.com/paper/2103.01748/) (2021)
- [An audio CAPTCHA to distinguish humans from computers](https://mlsanity.com/paper/1306.2885/) (2013)
- [Language Model Detoxification in Dialogue with Contextualized Stance Control](https://mlsanity.com/paper/2301.10368/) (2023)

## Built on by (most cited first, 26 of 26)

- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text](https://mlsanity.com/paper/2305.17359/) (2023)
- [Unbiased Watermark for Large Language Models](https://mlsanity.com/paper/2310.10669/) (2023)
- [Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models](https://mlsanity.com/paper/2311.04378/) (2023)
- [A Survey on Detection of LLMs-Generated Content](https://mlsanity.com/paper/2310.15654/) (2023)
- [AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://mlsanity.com/paper/2410.04265/) (2024)
- [Smaller Language Models are Better Black-box Machine-Generated Text Detectors](https://mlsanity.com/paper/2305.09859/) (2023)
- [Towards Possibilities & Impossibilities of AI-generated Text Detection: A Survey](https://mlsanity.com/paper/2310.15264/) (2023)
- [On the Detectability of ChatGPT Content: Benchmarking, Methodology, and Evaluation through the Lens of Academic Writing](https://mlsanity.com/paper/2306.05524/) (2023)
- [A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization](https://mlsanity.com/paper/2403.01152/) (2024)
- [Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Text](https://mlsanity.com/paper/2403.05750/) (2024)
- [Hidding the Ghostwriters: An Adversarial Evaluation of AI-Generated Student Essay Detection](https://mlsanity.com/paper/2402.00412/) (2024)
- [Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lacking](https://mlsanity.com/paper/2307.16806/) (2023)
- [Diversity Boosts AI-Generated Text Detection](https://mlsanity.com/paper/2509.18880/) (2025)
- [Are You Human? An Adversarial Benchmark to Expose LLMs](https://mlsanity.com/paper/2410.09569/) (2024)
- [Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models](https://mlsanity.com/paper/2504.01589/) (2025)
- [ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art](https://mlsanity.com/paper/2410.01733/) (2024)
- [Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams](https://mlsanity.com/paper/2512.16280/) (2025)
- [Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents](https://mlsanity.com/paper/2607.18659/) (2026)
- [InteractiveIE: Towards Assessing the Strength of Human-AI Collaboration in Improving the Performance of Information Extraction](https://mlsanity.com/paper/2305.14659/) (2023)
- [A blind spot for large language models: Supradiegetic linguistic information](https://mlsanity.com/paper/2306.06794/) (2023)
- [Unmasking the Imposters: How Censorship and Domain Adaptation Affect the Detection of Machine-Generated Tweets](https://mlsanity.com/paper/2406.17967/) (2024)
- [From Building Blocks to Planning: Multi-Step Spatial Reasoning in LLMs with Reinforcement Learning](https://mlsanity.com/paper/2512.24532/) (2025)
- [Random Initialization of Gated Sparse Adapters](https://mlsanity.com/paper/2511.01794/) (2025)
- [Breaking Darknet CAPTCHAs with general purpose LLM](https://mlsanity.com/paper/2608.28794/) (2026)

## Abstract (arXiv)

Large language models (LLMs) like GPT-4 have recently demonstrated impressive capabilities in natural language understanding and generation. However, there is a concern that they can be misused for malicious purposes, such as fraud or denial-of-service attacks. Therefore, it is crucial to develop methods for detecting whether the party involved in a conversation is a bot or a human. In this paper, we propose a framework named FLAIR, Finding Large Language Model Authenticity via a Single Inquiry and Response, to detect conversational bots in an online manner. Specifically, we target a single question scenario that can effectively differentiate human users from bots. The questions are divided into two categories: those that are easy for humans but difficult for bots (e.g., counting, substitution, searching, and ASCII art reasoning), and those that are easy for bots but difficult for humans (e.g., memorization and computation). Our approach shows different strengths of these questions in their effectiveness, providing a new way for online service providers to protect themselves against nefarious activities. Our code and question set are available at https://github.com/hongwang600/FLAIR.