# ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness

arXiv: https://arxiv.org/abs/2305.12947 · HTML: https://mlsanity.com/paper/2305.12947/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 22 of 22)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Mathematical Capabilities of ChatGPT](https://mlsanity.com/paper/2301.13867/) (2023)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT](https://mlsanity.com/paper/2302.10198/) (2023)
- [Semantic Parsing for Task Oriented Dialog using Hierarchical Representations](https://mlsanity.com/paper/1810.07942/) (2018)
- [ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning](https://mlsanity.com/paper/2304.06588/) (2023)
- [Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks](https://mlsanity.com/paper/2304.10145/) (2023)
- [Neural Syntactic Preordering for Controlled Paraphrase Generation](https://mlsanity.com/paper/2005.02013/) (2020)
- [Controllable Paraphrase Generation with a Syntactic Exemplar](https://mlsanity.com/paper/1906.00565/) (2019)
- [An Investigation of the (In)effectiveness of Counterfactually Augmented Data](https://mlsanity.com/paper/2107.00753/) (2021)
- [Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversity](https://mlsanity.com/paper/2008.04935/) (2020)
- [Outlier Detection for Improved Data Quality and Diversity in Dialog Systems](https://mlsanity.com/paper/1904.03122/) (2019)
- [Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt Tuning](https://mlsanity.com/paper/2202.00535/) (2022)
- [Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideation](https://mlsanity.com/paper/2101.06030/) (2021)
- [Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Features](https://mlsanity.com/paper/2111.03740/) (2021)

## Built on by (most cited first, 30 of 41)

- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [A Comprehensive Survey on Data Augmentation](https://mlsanity.com/paper/2405.09591/) (2024)
- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing](https://mlsanity.com/paper/2305.16635/) (2023)
- [A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation](https://mlsanity.com/paper/2401.09760/) (2024)
- [How Far Can We Extract Diverse Perspectives from Large Language Models?](https://mlsanity.com/paper/2311.09799/) (2023)
- [Authorship Obfuscation in Multilingual Machine-Generated Text Detection](https://mlsanity.com/paper/2401.07867/) (2024)
- [Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation](https://mlsanity.com/paper/2401.06643/) (2024)
- [Prompting a Large Language Model to Generate Diverse Motivational Messages: A Comparison with Human-Written Messages](https://mlsanity.com/paper/2308.13479/) (2023)
- [Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs](https://mlsanity.com/paper/2408.10902/) (2024)
- [Event-level Knowledge Editing](https://mlsanity.com/paper/2402.13093/) (2024)
- [What Has Been Lost with Synthetic Evaluation?](https://mlsanity.com/paper/2505.22830/) (2025)
- [Improving Black-box Robustness with In-Context Rewriting](https://mlsanity.com/paper/2402.08225/) (2024)
- [LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?](https://mlsanity.com/paper/2408.16502/) (2024)
- [Use Random Selection for Now: Investigation of Few-Shot Selection Strategies in LLM-based Text Augmentation for Classification](https://mlsanity.com/paper/2410.10756/) (2024)
- [Diversity-oriented Data Augmentation with Large Language Models](https://mlsanity.com/paper/2502.11671/) (2025)
- [AdParaphrase: Paraphrase Dataset for Analyzing Linguistic Features toward Generating Attractive Ad Texts](https://mlsanity.com/paper/2502.04674/) (2025)
- [Enterprise Large Language Model Evaluation Benchmark](https://mlsanity.com/paper/2506.20274/) (2025)
- [Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations](https://mlsanity.com/paper/2410.17099/) (2024)
- [A survey of diversity quantification in natural language processing: The why, what, where and how](https://mlsanity.com/paper/2507.20858/) (2025)
- [JurEE not Judges: safeguarding llm interactions with small, specialised Encoder Ensembles](https://mlsanity.com/paper/2410.08442/) (2024)
- [Aligning Sentence Simplification with ESL Learner's Proficiency for Language Acquisition](https://mlsanity.com/paper/2502.11457/) (2025)
- [A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languages](https://mlsanity.com/paper/2506.12158/) (2025)
- [Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification](https://mlsanity.com/paper/2403.05640/) (2024)
- [CookingSense: A Culinary Knowledgebase with Multidisciplinary Assertions](https://mlsanity.com/paper/2405.00523/) (2024)
- [Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolation](https://mlsanity.com/paper/2406.12471/) (2024)
- [AdParaphrase v2.0: Generating Attractive Ad Texts Using a Preference-Annotated Paraphrase Dataset](https://mlsanity.com/paper/2505.20826/) (2025)
- [Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth](https://mlsanity.com/paper/2506.06991/) (2025)
- [Improving the Distributional Alignment of LLMs using Supervision](https://mlsanity.com/paper/2507.00439/) (2025)
- [Overview of the Plagiarism Detection Task at PAN 2025](https://mlsanity.com/paper/2510.06805/) (2025)

## Abstract (arXiv)

The emergence of generative large language models (LLMs) raises the question: what will be its impact on crowdsourcing? Traditionally, crowdsourcing has been used for acquiring solutions to a wide variety of human-intelligence tasks, including ones involving text generation, modification or evaluation. For some of these tasks, models like ChatGPT can potentially substitute human workers. In this study, we investigate whether this is the case for the task of paraphrase generation for intent classification. We apply data collection methodology of an existing crowdsourcing study (similar scale, prompts and seed data) using ChatGPT and Falcon-40B. We show that ChatGPT-created paraphrases are more diverse and lead to at least as robust models.