# Human or Not? A Gamified Approach to the Turing Test

arXiv: https://arxiv.org/abs/2305.20010 · HTML: https://mlsanity.com/paper/2305.20010/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 7 references that are in the map.
- First posted on arXiv 31 May 2023.

## Builds on (most cited first, 1 of 1)

- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)

## Built on by (most cited first, 16 of 16)

- [People cannot distinguish GPT-4 from a human in a Turing test](https://mlsanity.com/paper/2405.08007/) (2024)
- [Large Language Models Pass the Turing Test](https://mlsanity.com/paper/2503.23674/) (2025)
- [How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHO](https://mlsanity.com/paper/2404.13957/) (2024)
- [Attributions toward Artificial Agents in a modified Moral Turing Test](https://mlsanity.com/paper/2406.11854/) (2024)
- [IMPersona: Evaluating Individual Level LM Impersonation](https://mlsanity.com/paper/2504.04332/) (2025)
- [International Governance of Civilian AI: A Jurisdictional Certification Approach](https://mlsanity.com/paper/2308.15514/) (2023)
- [X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents](https://mlsanity.com/paper/2408.09853/) (2024)
- [Are You Human? An Adversarial Benchmark to Expose LLMs](https://mlsanity.com/paper/2410.09569/) (2024)
- [When Incentives Backfire, Data Stops Being Human](https://mlsanity.com/paper/2502.07732/) (2025)
- [Survey of Consciousness Theory from Computational Perspective](https://mlsanity.com/paper/2309.10063/) (2023)
- [A Rigorous Turing Test: a Foundation for Evaluating Artificial General Intelligence](https://mlsanity.com/paper/2501.17629/) (2025)
- [How Open Must Language Models be to Enable Reliable Scientific Inference?](https://mlsanity.com/paper/2603.26539/) (2026)
- [ChatGPT-4 and other LLMs in the Turing Test: A Critical Analysis](https://mlsanity.com/paper/2503.06551/) (2025)
- [Using Large Language Models to Generate Authentic Multi-agent Knowledge Work Datasets](https://mlsanity.com/paper/2409.04286/) (2024)
- [Book your room in the Turing Hotel! A symmetric and distributed Turing Test with multiple AIs and humans](https://mlsanity.com/paper/2603.18981/) (2026)
- [Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignment](https://mlsanity.com/paper/2609.23640/) (2026)

## Abstract (arXiv)

We present "Human or Not?", an online game inspired by the Turing test, that measures the capability of AI chatbots to mimic humans in dialog, and of humans to tell bots from other humans. Over the course of a month, the game was played by over 1.5 million users who engaged in anonymous two-minute chat sessions with either another human or an AI language model which was prompted to behave like humans. The task of the players was to correctly guess whether they spoke to a person or to an AI. This largest scale Turing-style test conducted to date revealed some interesting facts. For example, overall users guessed the identity of their partners correctly in only 68% of the games. In the subset of the games in which users faced an AI bot, users had even lower correct guess rates of 60% (that is, not much higher than chance). This white paper details the development, deployment, and results of this unique experiment. While this experiment calls for many extensions and refinements, these findings already begin to shed light on the inevitable near future which will commingle humans and AI.