# Smaller Language Models are Better Black-box Machine-Generated Text Detectors

arXiv: https://arxiv.org/abs/2305.09859 · HTML: https://mlsanity.com/paper/2305.09859/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 26 of 26)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature](https://mlsanity.com/paper/2301.11305/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense](https://mlsanity.com/paper/2303.13408/) (2023)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [Can AI-Generated Text be Reliably Detected?](https://mlsanity.com/paper/2303.11156/) (2023)
- [Automatic Detection of Generated Text is Easiest when Humans are Fooled](https://mlsanity.com/paper/1911.00650/) (2019)
- [Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://mlsanity.com/paper/2203.03929/) (2022)
- [Automatic Detection of Machine Generated Text: A Critical Survey](https://mlsanity.com/paper/2011.01314/) (2020)
- [Contextualized Perturbation for Textual Adversarial Attack](https://mlsanity.com/paper/2009.07502/) (2020)
- [Adversarial Watermarking Transformer: Towards Tracing Text Provenance with Data Hiding](https://mlsanity.com/paper/2009.03015/) (2020)
- [TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation](https://mlsanity.com/paper/2109.13296/) (2021)
- [Real or Fake? Learning to Discriminate Machine from Human Generated Text](https://mlsanity.com/paper/1906.03351/) (2019)
- [MGTBench: Benchmarking Machine-Generated Text Detection](https://mlsanity.com/paper/2303.14822/) (2023)
- [Neural Linguistic Steganography](https://mlsanity.com/paper/1909.01496/) (2019)
- [Real or Fake Text?: Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text](https://mlsanity.com/paper/2212.12672/) (2022)
- [Generating Steganographic Text with LSTMs](https://mlsanity.com/paper/1705.10742/) (2017)
- [Bot or Human? Detecting ChatGPT Imposters with A Single Question](https://mlsanity.com/paper/2305.06424/) (2023)

## Built on by (most cited first, 30 of 35)

- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [Can LLM-Generated Misinformation Be Detected?](https://mlsanity.com/paper/2309.13788/) (2023)
- [DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text](https://mlsanity.com/paper/2305.17359/) (2023)
- [A Survey on Detection of LLMs-Generated Content](https://mlsanity.com/paper/2310.15654/) (2023)
- [An Unforgeable Publicly Verifiable Watermark for Large Language Models](https://mlsanity.com/paper/2307.16230/) (2023)
- [Raidar: geneRative AI Detection viA Rewriting](https://mlsanity.com/paper/2401.12970/) (2024)
- [Large Language Models can be Guided to Evade AI-Generated Text Detection](https://mlsanity.com/paper/2305.10847/) (2023)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trick](https://mlsanity.com/paper/2402.12948/) (2024)
- [Technical Report on the Pangram AI-Generated Text Classifier](https://mlsanity.com/paper/2402.14873/) (2024)
- [Towards Possibilities & Impossibilities of AI-generated Text Detection: A Survey](https://mlsanity.com/paper/2310.15264/) (2023)
- [Between Lines of Code: Unraveling the Distinct Patterns of Machine and Human Programmers](https://mlsanity.com/paper/2401.06461/) (2024)
- [Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attack](https://mlsanity.com/paper/2404.01907/) (2024)
- [From Text to Source: Results in Detecting Large Language Model-Generated Content](https://mlsanity.com/paper/2309.13322/) (2023)
- [Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks](https://mlsanity.com/paper/2402.11638/) (2024)
- [Zero-Shot Detection of Machine-Generated Codes](https://mlsanity.com/paper/2310.05103/) (2023)
- [Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection](https://mlsanity.com/paper/2412.11506/) (2024)
- [EAGLE: A Domain Generalization Framework for AI-generated Text Detection](https://mlsanity.com/paper/2403.15690/) (2024)
- [DALD: Improving Logits-based Detector without Logits from Black-box LLMs](https://mlsanity.com/paper/2406.05232/) (2024)
- [On the Zero-Shot Generalization of Machine-Generated Text Detectors](https://mlsanity.com/paper/2310.05165/) (2023)
- [Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better](https://mlsanity.com/paper/2402.00263/) (2024)
- [Duwak: Dual Watermarks in Large Language Models](https://mlsanity.com/paper/2403.13000/) (2024)
- [Hidding the Ghostwriters: An Adversarial Evaluation of AI-Generated Student Essay Detection](https://mlsanity.com/paper/2402.00412/) (2024)
- [A Comprehensive Analysis of Large Language Model Outputs: Similarity, Diversity, and Bias](https://mlsanity.com/paper/2505.09056/) (2025)
- [Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors](https://mlsanity.com/paper/2312.12918/) (2023)
- [Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection](https://mlsanity.com/paper/2510.08602/) (2025)
- [The Impact of Prompts on Zero-Shot Detection of AI-Generated Text](https://mlsanity.com/paper/2403.20127/) (2024)
- [Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity](https://mlsanity.com/paper/2407.13105/) (2024)
- [LSHBloom: Memory-efficient, Extreme-scale Document Deduplication](https://mlsanity.com/paper/2411.04257/) (2024)
- [Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection](https://mlsanity.com/paper/2501.11786/) (2025)

## Abstract (arXiv)

With the advent of fluent generative language models that can produce convincing utterances very similar to those written by humans, distinguishing whether a piece of text is machine-generated or human-written becomes more challenging and more important, as such models could be used to spread misinformation, fake news, fake reviews and to mimic certain authors and figures. To this end, there have been a slew of methods proposed to detect machine-generated text. Most of these methods need access to the logits of the target model or need the ability to sample from the target. One such black-box detection method relies on the observation that generated text is locally optimal under the likelihood function of the generator, while human-written text is not. We find that overall, smaller and partially-trained models are better universal text detectors: they can more precisely detect text generated from both small and larger models. Interestingly, we find that whether the detector and generator were trained on the same data is not critically important to the detection success. For instance the OPT-125M model has an AUC of 0.81 in detecting ChatGPT generations, whereas a larger model from the GPT family, GPTJ-6B, has AUC of 0.45.