SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU
Not on the map. Why?
Abstract (arXiv)
Task-oriented dialogue (ToD) systems help users execute well-defined tasks across a variety of domains (e.g., textitflight booking or textitfood ordering), with their Natural Language Understanding (NLU) components being dedicated to the analysis of user utterances, predicting users' intents (textitIntent Detection, ID) and extracting values for informational slots (textitValue Extraction, VE). In most domains, labelled NLU data is scarce, making sample-efficient learning -- enabled with effective transfer paradigms -- paramount. In this work, we introduce SQATIN, a new framework for dialog NLU based on (i) instruction tuning and (ii) question-answering-based formulation of ID and VE tasks. According to the evaluation on established NLU benchmarks, SQATIN sets the new state of the art in dialogue NLU, substantially surpassing the performance of current models based on standard fine-tuning objectives in both in-domain training and cross-domain transfer. SQATIN yields particularly large performance gains in cross-domain transfer, owing to the fact that our QA-based instruction tuning leverages similarities between natural language descriptions of classes (i.e., slots and intents) across domains.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 51 of its 66 references that are in the map. First posted on arXiv 16 Nov 2023.
Builds on
51 of 66 references.
References · 51
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
- 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
- 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
- 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
- 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
- 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
- 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
- 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
- 2020Language-agnostic BERT Sentence Embeddingcited by 621
- 2020Towards a Human-like Open-Domain Chatbotcited by 467
- 2021MetaICL: Learning to Learn In Contextcited by 450
- 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
- 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
- 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
- 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
- 2021Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue Systemcited by 139
- 2019Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systemscited by 110
- 2020MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systemscited by 103
- 2020PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learningcited by 79
- 2021Few-Shot Bot: Prompt-Based Learning for Dialogue Systemscited by 64
- 2019Dialog State Tracking: A Neural Reading Comprehension Approachcited by 62
- 2021Zero-Shot Dialogue State Tracking via Cross-Task Transfercited by 52
- 2023Are LLMs All You Need for Task-Oriented Dialogue?cited by 52
- 2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35
- 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
- 2021CINS: Comprehensive Instruction for Few-shot Learning in Task-oriented Dialog Systemscited by 30
- 2022Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialogcited by 30
- 2020Language Model is All You Need: Natural Language Understanding as Question Answeringcited by 29
- 2021Pretraining the Noisy Channel Model for Task-Oriented Dialoguecited by 23
- 2020ConVEx: Data-Efficient and Few-Shot Slot Labelingcited by 22
- 2021DS-TOD: Efficient Domain Specialization for Task Oriented Dialogcited by 21
- 2021Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systemscited by 18
- 2022MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialoguecited by 15
- 2020Learning a Simple and Effective Model for Multi-turn Response Generation with Auxiliary Taskscited by 14
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.