# Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations

arXiv: https://arxiv.org/abs/2305.14556 · HTML: https://mlsanity.com/paper/2305.14556/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 67 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Is GPT-3 a Good Data Annotator?](https://mlsanity.com/paper/2212.10450/) (2022)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation](https://mlsanity.com/paper/2104.00773/) (2021)
- [Recent Neural Methods on Slot Filling and Intent Classification for Task-Oriented Dialogue Systems: A Survey](https://mlsanity.com/paper/2011.00564/) (2020)
- [Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances](https://mlsanity.com/paper/2204.03375/) (2022)
- ["Mama Always Had a Way of Explaining Things So I Could Understand'': A Dialogue Corpus for Learning to Construct Explanations](https://mlsanity.com/paper/2209.02508/) (2022)

## Built on by (most cited first, 14 of 14)

- [Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges](https://mlsanity.com/paper/2403.02990/) (2024)
- [GPTs Are Multilingual Annotators for Sequence Generation Tasks](https://mlsanity.com/paper/2402.05512/) (2024)
- [Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis](https://mlsanity.com/paper/2309.15656/) (2023)
- [GPT Self-Supervision for a Better Data Annotator](https://mlsanity.com/paper/2306.04349/) (2023)
- [Stephanie: Step-by-Step Dialogues for Mimicking Human Interactions in Social Conversations](https://mlsanity.com/paper/2407.04093/) (2024)
- [TOD-Flow: Modeling the Structure of Task-Oriented Dialogues](https://mlsanity.com/paper/2312.04668/) (2023)
- [An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analyses](https://mlsanity.com/paper/2503.21393/) (2025)
- [SPADE: Structured Prompting Augmentation for Dialogue Enhancement in Machine-Generated Text Detection](https://mlsanity.com/paper/2503.15044/) (2025)
- [Positional Bias in Binary Question Answering: How Uncertainty Shapes Model Preferences](https://mlsanity.com/paper/2506.23743/) (2025)
- [Training Zero-Shot Generalizable End-to-End Task-Oriented Dialog System Without Turn-level Dialog Annotations](https://mlsanity.com/paper/2407.15055/) (2024)
- [Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback](https://mlsanity.com/paper/2502.13298/) (2025)
- [Text Production and Comprehension by Human and Artificial Intelligence: Interdisciplinary Workshop Report](https://mlsanity.com/paper/2506.22698/) (2025)
- [Codebook-Injected Dialogue Segmentation for Multi-Utterance Constructs Annotation: LLM-Assisted and Gold-Label-Free Evaluation](https://mlsanity.com/paper/2601.12061/) (2026)
- [Domain-Adapted Retrieval for In-Context Annotation of Pedagogical Dialogue Acts](https://mlsanity.com/paper/2604.03127/) (2026)

## Abstract (arXiv)

Large pre-trained language models have exhibited unprecedented capabilities in producing high-quality text via prompting techniques. This fact introduces new possibilities for data collection and annotation, particularly in situations where such data is scarce, complex to gather, expensive, or even sensitive. In this paper, we explore the potential of these models to generate and annotate goal-oriented dialogues, and conduct an in-depth analysis to evaluate their quality. Our experiments employ ChatGPT, and encompass three categories of goal-oriented dialogues (task-oriented, collaborative, and explanatory), two generation modes (interactive and one-shot), and two languages (English and Italian). Based on extensive human-based evaluations, we demonstrate that the quality of generated dialogues and annotations is on par with those generated by humans.