# Detecting The Corruption Of Online Questionnaires By Artificial Intelligence

arXiv: https://arxiv.org/abs/2308.07499 · HTML: https://mlsanity.com/paper/2308.07499/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 112 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection](https://mlsanity.com/paper/2301.07597/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [ChatGPT: The End of Online Exam Integrity?](https://mlsanity.com/paper/2212.09292/) (2022)
- [ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text](https://mlsanity.com/paper/2301.13852/) (2023)
- [Stylometric Detection of AI-Generated Text in Twitter Timelines](https://mlsanity.com/paper/2303.03697/) (2023)
- [ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing](https://mlsanity.com/paper/2303.13367/) (2023)
- [Putting ChatGPT's Medical Advice to the (Turing) Test](https://mlsanity.com/paper/2301.10035/) (2023)
- [To ChatGPT, or not to ChatGPT: That is the question!](https://mlsanity.com/paper/2304.01487/) (2023)
- [ChatGPT Participates in a Computer Science Exam](https://mlsanity.com/paper/2303.09461/) (2023)
- [Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions](https://mlsanity.com/paper/1801.02546/) (2018)
- [Comparison of Quality Indicators in User-generated Content Using Social Media and Scholarly Text](https://mlsanity.com/paper/1910.11399/) (2019)

## Built on by (most cited first, 3 of 3)

- [A Rigorous Turing Test: a Foundation for Evaluating Artificial General Intelligence](https://mlsanity.com/paper/2501.17629/) (2025)
- [From Noise to Insights: Enhancing Supply Chain Decision Support through AI-Based Survey Integrity Analytics](https://mlsanity.com/paper/2601.17005/) (2026)
- [Cheap, open agents make LLM pollution harder to mitigate](https://mlsanity.com/paper/2609.31054/) (2026)

## Abstract (arXiv)

Online questionnaires that use crowd-sourcing platforms to recruit participants have become commonplace, due to their ease of use and low costs. Artificial Intelligence (AI) based Large Language Models (LLM) have made it easy for bad actors to automatically fill in online forms, including generating meaningful text for open-ended tasks. These technological advances threaten the data quality for studies that use online questionnaires. This study tested if text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. While humans were able to correctly identify authorship of text above chance level (76 percent accuracy), their performance was still below what would be required to ensure satisfactory data quality. Researchers currently have to rely on the disinterest of bad actors to successfully use open-ended responses as a useful tool for ensuring data quality. Automatic AI detection systems are currently completely unusable. If AIs become too prevalent in submitting responses then the costs associated with detecting fraudulent submissions will outweigh the benefits of online questionnaires. Individual attention checks will no longer be a sufficient tool to ensure good data quality. This problem can only be systematically addressed by crowd-sourcing platforms. They cannot rely on automatic AI detection systems and it is unclear how they can ensure data quality for their paying clients.