mlsanityData to 7 Oct 2026

ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness

arXiv 2305.12947 ↗First posted 22 May 2023cs.CLcited by 41

Not on the map. Why?

Abstract (arXiv)

The emergence of generative large language models (LLMs) raises the question: what will be its impact on crowdsourcing? Traditionally, crowdsourcing has been used for acquiring solutions to a wide variety of human-intelligence tasks, including ones involving text generation, modification or evaluation. For some of these tasks, models like ChatGPT can potentially substitute human workers. In this study, we investigate whether this is the case for the task of paraphrase generation for intent classification. We apply data collection methodology of an existing crowdsourcing study (similar scale, prompts and seed data) using ChatGPT and Falcon-40B. We show that ChatGPT-created paraphrases are more diverse and lead to at least as robust models.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 22 of its 35 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

22 of 35 references.

References · 22

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  3. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  4. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  5. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  6. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  7. 2023Mathematical Capabilities of ChatGPTcited by 263
  8. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 181
  9. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  10. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  11. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  12. 2018Semantic Parsing for Task Oriented Dialog using Hierarchical Representationscited by 101
10 more
  1. 2023ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learningcited by 97
  2. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  3. 2020Neural Syntactic Preordering for Controlled Paraphrase Generationcited by 56
  4. 2019Controllable Paraphrase Generation with a Syntactic Exemplarcited by 44
  5. 2021An Investigation of the (In)effectiveness of Counterfactually Augmented Datacited by 42
  6. 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
  7. 2019Outlier Detection for Improved Data Quality and Diversity in Dialog Systemscited by 28
  8. 2022Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt Tuningcited by 25
  9. 2021Directed Diversity: Leveraging Language Embedding Distances for Collective Creativity in Crowd Ideationcited by 23
  10. 2021Toward Learning Human-aligned Cross-domain Robust Models by Countering Misaligned Featurescited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 20; 2025: 14; 2026: 3.

41 papers.

Sort:

By area:

  1. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  2. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  3. 2024A Comprehensive Survey on Data Augmentationcited by 38
  4. 2023Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasingcited by 31
  5. 2024A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregationcited by 23
  6. 2023How Far Can We Extract Diverse Perspectives from Large Language Models?cited by 22
  7. 2024Authorship Obfuscation in Multilingual Machine-Generated Text Detectioncited by 19
  8. 2024Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentationcited by 17
  9. 2023Prompting a Large Language Model to Generate Diverse Motivational Messages: A Comparison with Human-Written Messagescited by 10
  10. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  11. 2024Improving Black-box Robustness with In-Context Rewritingcited by 8
  12. 2024Event-level Knowledge Editingcited by 8
29 more
  1. 2024LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?cited by 8
  2. 2025What Has Been Lost with Synthetic Evaluation?cited by 8
  3. 2024Use Random Selection for Now: Investigation of Few-Shot Selection Strategies in LLM-based Text Augmentation for Classificationcited by 7
  4. 2025Diversity-oriented Data Augmentation with Large Language Modelscited by 7
  5. 2025AdParaphrase: Paraphrase Dataset for Analyzing Linguistic Features toward Generating Attractive Ad Textscited by 5
  6. 2025Enterprise Large Language Model Evaluation Benchmarkcited by 5
  7. 2024Human-LLM Hybrid Text Answer Aggregation for Crowd Annotationscited by 4
  8. 2025A survey of diversity quantification in natural language processing: The why, what, where and howcited by 4
  9. 2024JurEE not Judges: safeguarding llm interactions with small, specialised Encoder Ensemblescited by 3
  10. 2025Aligning Sentence Simplification with ESL Learner's Proficiency for Language Acquisitioncited by 3
  11. 2024Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classificationcited by 2
  12. 2024CookingSense: A Culinary Knowledgebase with Multidisciplinary Assertionscited by 2
  13. 2024Fighting Randomness with Randomness: Mitigating Optimisation Instability of Fine-Tuning using Delayed Ensemble and Noisy Interpolationcited by 2
  14. 2025AdParaphrase v2.0: Generating Attractive Ad Texts Using a Preference-Annotated Paraphrase Datasetcited by 2
  15. 2025Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truthcited by 2
  16. 2025A Rigorous Evaluation of LLM Data Generation Strategies for Low-Resource Languagescited by 2
  17. 2025Improving the Distributional Alignment of LLMs using Supervisioncited by 2
  18. 2025Overview of the Plagiarism Detection Task at PAN 2025cited by 2
  19. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classificationcited by 2
  20. 2024ConFit: Improving Resume-Job Matching using Data Augmentation and Contrastive Learningcited by 1
  21. 2025RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Setscited by 1
  22. 2025Prompt Optimization Across Multiple Agents for Representing Diverse Human Populationscited by 1
  23. 2026Generation-Based and Emotion-Reflected Memory Update: Creating the KEEM Dataset for Better Long-Term Conversationcited by 1
  24. 2023Keeping Users Engaged During Repeated Administration of the Same Questionnaire: Using Large Language Models to Reliably Diversify Questionscited by 0
  25. 2024Incorporating Different Verbal Cues to Improve Text-Based Computer-Delivered Health Messagingcited by 0
  26. 2024A Survey on the Real Power of ChatGPTcited by 0
  27. 2024What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicingcited by 0
  28. 2025Are Large Language Models the future crowd workers of Linguistics?cited by 0
  29. 2026Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generationcited by 0