# LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Tagging

arXiv: https://arxiv.org/abs/2209.09900 · HTML: https://mlsanity.com/paper/2209.09900/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 54 references that are in the map.
- First posted on arXiv 20 Sep 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [End-to-End Slot Alignment and Recognition for Cross-Lingual NLU](https://mlsanity.com/paper/2004.14353/) (2020)
- [An Analysis of Simple Data Augmentation for Named Entity Recognition](https://mlsanity.com/paper/2010.11683/) (2020)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model](https://mlsanity.com/paper/2208.01448/) (2022)
- [Neural Data Augmentation via Example Extrapolation](https://mlsanity.com/paper/2102.01335/) (2021)
- [Neural Paraphrase Generation with Stacked Residual LSTM Networks](https://mlsanity.com/paper/1610.03098/) (2016)
- [SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup](https://mlsanity.com/paper/2010.02322/) (2020)
- [Data Augmentation for Intent Classification with Off-the-shelf Large Language Models](https://mlsanity.com/paper/2204.01959/) (2022)
- [Controlled Text Generation for Data Augmentation in Intelligent Artificial Agents](https://mlsanity.com/paper/1910.03487/) (2019)

## Built on by (most cited first, 29 of 29)

- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [MIMIC-IT: Multi-Modal In-Context Instruction Tuning](https://mlsanity.com/paper/2306.05425/) (2023)
- [PLACES: Prompting Language Models for Social Conversation Synthesis](https://mlsanity.com/paper/2302.03269/) (2023)
- [FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering](https://mlsanity.com/paper/2308.12060/) (2023)
- [Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers](https://mlsanity.com/paper/2404.04925/) (2024)
- [Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models](https://mlsanity.com/paper/2410.11718/) (2024)
- [CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing](https://mlsanity.com/paper/2210.07074/) (2022)
- [LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Text](https://mlsanity.com/paper/2402.04335/) (2024)
- [Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-tuning](https://mlsanity.com/paper/2410.08081/) (2024)
- [FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data](https://mlsanity.com/paper/2408.06273/) (2024)
- [Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models](https://mlsanity.com/paper/2305.12544/) (2023)
- [Toward Unified Controllable Text Generation via Regular Expression Instruction](https://mlsanity.com/paper/2309.10447/) (2023)
- [Grounding Data Science Code Generation with Input-Output Specifications](https://mlsanity.com/paper/2402.08073/) (2024)
- [Enhancing Low-Resource LLMs Classification with PEFT and Synthetic Data](https://mlsanity.com/paper/2404.02422/) (2024)
- [Language Models are Causal Knowledge Extractors for Zero-shot Video Question Answering](https://mlsanity.com/paper/2304.03754/) (2023)
- [PersonalityChat: Conversation Distillation for Personalized Dialog Modeling with Facts and Traits](https://mlsanity.com/paper/2401.07363/) (2024)
- [Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training](https://mlsanity.com/paper/2306.05278/) (2023)
- [ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Filler](https://mlsanity.com/paper/2403.17536/) (2024)
- [KwaiYiiMath: Technical Report](https://mlsanity.com/paper/2310.07488/) (2023)
- [Decoupling Representation and Knowledge for Few-Shot Intent Classification and Slot Filling](https://mlsanity.com/paper/2312.13495/) (2023)
- [Generating Hard-Negative Out-of-Scope Data with ChatGPT for Intent Classification](https://mlsanity.com/paper/2403.05640/) (2024)
- [Evaluating Inter-Bilingual Semantic Parsing for Indian Languages](https://mlsanity.com/paper/2304.13005/) (2023)
- [Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models](https://mlsanity.com/paper/2306.08756/) (2023)
- [Better Python Programming for all: With the focus on Maintainability](https://mlsanity.com/paper/2408.09134/) (2024)
- [On the Necessity of World Knowledge for Mitigating Missing Labels in Extreme Classification](https://mlsanity.com/paper/2408.09585/) (2024)
- [CALICO: Conversational Agent Localization via Synthetic Data Generation](https://mlsanity.com/paper/2412.05388/) (2024)
- [Integrating gender inclusivity into large language models via instruction tuning](https://mlsanity.com/paper/2508.18466/) (2025)
- [Language-Aware Token Boosting: LLM Language Confusion Reduction Without Tuning](https://mlsanity.com/paper/2606.08994/) (2026)

## Abstract (arXiv)

We present LINGUIST, a method for generating annotated data for Intent Classification and Slot Tagging (IC+ST), via fine-tuning AlexaTM 5B, a 5-billion-parameter multilingual sequence-to-sequence (seq2seq) model, on a flexible instruction prompt. In a 10-shot novel intent setting for the SNIPS dataset, LINGUIST surpasses state-of-the-art approaches (Back-Translation and Example Extrapolation) by a wide margin, showing absolute improvement for the target intents of +1.9 points on IC Recall and +2.5 points on ST F1 Score. In the zero-shot cross-lingual setting of the mATIS++ dataset, LINGUIST out-performs a strong baseline of Machine Translation with Slot Alignment by +4.14 points absolute on ST F1 Score across 6 languages, while matching performance on IC. Finally, we verify our results on an internal large-scale multilingual dataset for conversational agent IC+ST and show significant improvements over a baseline which uses Back-Translation, Paraphrasing and Slot Catalog Resampling. To our knowledge, we are the first to demonstrate instruction fine-tuning of a large-scale seq2seq model to control the outputs of multilingual intent- and slot-labeled data generation.