# A Preliminary Evaluation of ChatGPT for Zero-shot Dialogue Understanding

arXiv: https://arxiv.org/abs/2304.04256 · HTML: https://mlsanity.com/paper/2304.04256/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 9 Apr 2023.

## Builds on (most cited first, 18 of 18)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization](https://mlsanity.com/paper/2302.08081/) (2023)
- [MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation](https://mlsanity.com/paper/2104.00773/) (2021)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [A Co-Interactive Transformer for Joint Slot Filling and Intent Detection](https://mlsanity.com/paper/2010.03880/) (2020)
- [A Survey on Spoken Language Understanding: Recent Advances and New Frontiers](https://mlsanity.com/paper/2103.03095/) (2021)
- [Description-Driven Task-Oriented Dialog Modeling](https://mlsanity.com/paper/2201.08904/) (2022)
- [Coach: A Coarse-to-Fine Approach for Cross-domain Slot Filling](https://mlsanity.com/paper/2004.11727/) (2020)
- ["Do you follow me?": A Survey of Recent Approaches in Dialogue State Tracking](https://mlsanity.com/paper/2207.14627/) (2022)

## Built on by (most cited first, 30 of 31)

- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought](https://mlsanity.com/paper/2410.05695/) (2024)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models](https://mlsanity.com/paper/2305.13112/) (2023)
- [Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation](https://mlsanity.com/paper/2403.09738/) (2024)
- [Exploring the Potential of Large Language Models in Computational Argumentation](https://mlsanity.com/paper/2311.09022/) (2023)
- [Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistance](https://mlsanity.com/paper/2305.13225/) (2023)
- [What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration](https://mlsanity.com/paper/2410.20482/) (2024)
- [Large Language Models are Complex Table Parsers](https://mlsanity.com/paper/2312.11521/) (2023)
- [Can Small Language Models be Good Reasoners for Sequential Recommendation?](https://mlsanity.com/paper/2403.04260/) (2024)
- [Towards LLM-driven Dialogue State Tracking](https://mlsanity.com/paper/2310.14970/) (2023)
- [Balancing Specialized and General Skills in LLMs: The Impact of Modern Tuning and Data Strategy](https://mlsanity.com/paper/2310.04945/) (2023)
- [End-to-end Task-oriented Dialogue: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2311.09008/) (2023)
- [Beyond the Known: Investigating LLMs Performance on Out-of-Domain Intent Detection](https://mlsanity.com/paper/2402.17256/) (2024)
- [RBF++: Quantifying and Optimizing Reasoning Boundaries across Measurable and Unmeasurable Capabilities for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2505.13307/) (2025)
- [Speech-based Slot Filling using Large Language Models](https://mlsanity.com/paper/2311.07418/) (2023)
- [CroPrompt: Cross-task Interactive Prompting for Zero-shot Spoken Language Understanding](https://mlsanity.com/paper/2406.10505/) (2024)
- [LANID: LLM-assisted New Intent Discovery](https://mlsanity.com/paper/2503.23740/) (2025)
- [DMDTEval: An Evaluation and Analysis of LLMs on Disambiguation in Multi-domain Translation](https://mlsanity.com/paper/2504.20371/) (2025)
- [A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding](https://mlsanity.com/paper/2409.15861/) (2024)
- [LLM-Assisted Automated Deductive Coding of Dialogue Data: Leveraging Dialogue-Specific Characteristics to Enhance Contextual Understanding](https://mlsanity.com/paper/2504.19734/) (2025)
- [ICE-GRT: Instruction Context Enhancement by Generative Reinforcement based Transformers](https://mlsanity.com/paper/2401.02072/) (2024)
- [Gradable ChatGPT Translation Evaluation](https://mlsanity.com/paper/2401.09984/) (2024)
- [Word Order's Impacts: Insights from Reordering and Generation Analysis](https://mlsanity.com/paper/2403.11473/) (2024)
- [Benchmarks Underestimate the Readiness of Multi-lingual Dialogue Agents](https://mlsanity.com/paper/2405.17840/) (2024)
- [Introducing Visual Scenes and Reasoning: A More Realistic Benchmark for Spoken Language Understanding](https://mlsanity.com/paper/2511.19005/) (2025)
- [Bridging Reasoning and Action: Hybrid LLM-RL Framework for Efficient Cross-Domain Task-Oriented Dialogue](https://mlsanity.com/paper/2604.23345/) (2026)
- [MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models](https://mlsanity.com/paper/2606.07996/) (2026)

## Abstract (arXiv)

Zero-shot dialogue understanding aims to enable dialogue to track the user's needs without any training data, which has gained increasing attention. In this work, we investigate the understanding ability of ChatGPT for zero-shot dialogue understanding tasks including spoken language understanding (SLU) and dialogue state tracking (DST). Experimental results on four popular benchmarks reveal the great potential of ChatGPT for zero-shot dialogue understanding. In addition, extensive analysis shows that ChatGPT benefits from the multi-turn interactive prompt in the DST task but struggles to perform slot filling for SLU. Finally, we summarize several unexpected behaviors of ChatGPT in dialogue understanding tasks, hoping to provide some insights for future research on building zero-shot dialogue understanding systems with Large Language Models (LLMs).