# A Statistical Turing Test for Generative Models

arXiv: https://arxiv.org/abs/2309.08913 · HTML: https://mlsanity.com/paper/2309.08913/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 46 references that are in the map.
- First posted on arXiv 16 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [Automatic Detection of Generated Text is Easiest when Humans are Fooled](https://mlsanity.com/paper/1911.00650/) (2019)
- [Media Forensics and DeepFakes: an overview](https://mlsanity.com/paper/2001.06564/) (2020)
- [A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity](https://mlsanity.com/paper/2305.13169/) (2023)
- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://mlsanity.com/paper/2303.10130/) (2023)
- [Deep Learning for Deepfakes Creation and Detection: A Survey](https://mlsanity.com/paper/1909.11573/) (2019)
- [Real or Fake? Learning to Discriminate Machine from Human Generated Text](https://mlsanity.com/paper/1906.03351/) (2019)
- [Comparing Foundation Models using Data Kernels](https://mlsanity.com/paper/2305.05126/) (2023)
- [A partition-based similarity for classification distributions](https://mlsanity.com/paper/2011.06557/) (2020)

## Built on by (most cited first, 8 of 8)

- [Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text](https://mlsanity.com/paper/2401.12070/) (2024)
- [Tracking the perspectives of interacting language models](https://mlsanity.com/paper/2406.11938/) (2024)
- [Statistical inference on black-box generative models in the data kernel perspective space](https://mlsanity.com/paper/2410.01106/) (2024)
- [X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents](https://mlsanity.com/paper/2408.09853/) (2024)
- [Investigating social alignment via mirroring in a system of interacting language models](https://mlsanity.com/paper/2412.06834/) (2024)
- [Toward a digital twin of U.S. Congress](https://mlsanity.com/paper/2505.00006/) (2025)
- [Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations](https://mlsanity.com/paper/2509.10963/) (2025)
- [When prompt perturbations break your A/B test: A valid statistical test for generative surveying](https://mlsanity.com/paper/2605.27463/) (2026)

## Abstract (arXiv)

The emergence of human-like abilities of AI systems for content generation in domains such as text, audio, and vision has prompted the development of classifiers to determine whether content originated from a human or a machine. Implicit in these efforts is an assumption that the generation properties of a human are different from that of the machine. In this work, we provide a framework in the language of statistical pattern recognition that quantifies the difference between the distributions of human and machine-generated content conditioned on an evaluation context. We describe current methods in the context of the framework and demonstrate how to use the framework to evaluate the progression of generative models towards human-like capabilities, among many axes of analysis.