# Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understanding

arXiv: https://arxiv.org/abs/2305.13512 · HTML: https://mlsanity.com/paper/2305.13512/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 22 May 2023.

## Builds on (most cited first, 30 of 37)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network](https://mlsanity.com/paper/2006.05702/) (2020)
- [A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding](https://mlsanity.com/paper/2111.02735/) (2021)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [A Survey on Spoken Language Understanding: Recent Advances and New Frontiers](https://mlsanity.com/paper/2103.03095/) (2021)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [Multilingual and Cross-Lingual Intent Detection from Spoken Data](https://mlsanity.com/paper/2104.08524/) (2021)
- [SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks](https://mlsanity.com/paper/2303.00733/) (2023)

## Built on by (most cited first, 21 of 21)

- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech](https://mlsanity.com/paper/2309.09510/) (2023)
- [On the (In)Effectiveness of Large Language Models for Chinese Text Correction](https://mlsanity.com/paper/2307.09007/) (2023)
- [Can Whisper perform speech-based in-context learning?](https://mlsanity.com/paper/2309.07081/) (2023)
- [Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learning](https://mlsanity.com/paper/2310.04782/) (2023)
- [Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs](https://mlsanity.com/paper/2502.14830/) (2025)
- [Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyond](https://mlsanity.com/paper/2402.14259/) (2024)
- [Leveraging Large Language Models for Exploiting ASR Uncertainty](https://mlsanity.com/paper/2309.04842/) (2023)
- [Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets](https://mlsanity.com/paper/2405.02132/) (2024)
- [Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks](https://mlsanity.com/paper/2401.02921/) (2024)
- [CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems](https://mlsanity.com/paper/2604.12312/) (2026)
- [Towards interfacing large language models with ASR systems using confidence measures and prompting](https://mlsanity.com/paper/2407.21414/) (2024)
- [Speech Understanding on Tiny Devices with A Learning Cache](https://mlsanity.com/paper/2311.18188/) (2023)
- [CroPrompt: Cross-task Interactive Prompting for Zero-shot Spoken Language Understanding](https://mlsanity.com/paper/2406.10505/) (2024)
- [Spoken Language Understanding on Unseen Tasks With In-Context Learning](https://mlsanity.com/paper/2505.07731/) (2025)
- [Do Large Language Models Need Intent? Revisiting Response Generation Strategies for Service Assistant](https://mlsanity.com/paper/2509.05006/) (2025)
- [MAC-SLU: Multi-Intent Automotive Cabin Spoken Language Understanding Benchmark](https://mlsanity.com/paper/2512.01603/) (2025)
- [Leveraging Information Retrieval to Enhance Spoken Language Understanding Prompts in Few-Shot Learning](https://mlsanity.com/paper/2506.03035/) (2025)
- [Assessing the feasibility of Large Language Models for detecting micro-behaviors in team interactions during space missions](https://mlsanity.com/paper/2506.22679/) (2025)
- [A Computational Approach to Modeling Conversational Systems: Analyzing Large-Scale Quasi-Patterned Dialogue Flows](https://mlsanity.com/paper/2507.13544/) (2025)
- [Retrieval-Augmented Generation in LLMs for Mental Health: Quantifying the Incremental Contribution of Retrieval Within a Layered Safety Architecture](https://mlsanity.com/paper/2607.24817/) (2026)

## Abstract (arXiv)

Recently, large pretrained language models have demonstrated strong language understanding capabilities. This is particularly reflected in their zero-shot and in-context learning abilities on downstream tasks through prompting. To assess their impact on spoken language understanding (SLU), we evaluate several such models like ChatGPT and OPT of different sizes on multiple benchmarks. We verify the emergent ability unique to the largest models as they can reach intent classification accuracy close to that of supervised models with zero or few shots on various languages given oracle transcripts. By contrast, the results for smaller models fitting a single GPU fall far behind. We note that the error cases often arise from the annotation scheme of the dataset; responses from ChatGPT are still reasonable. We show, however, that the model is worse at slot filling, and its performance is sensitive to ASR errors, suggesting serious challenges for the application of those textual models on SLU.