# Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning

arXiv: https://arxiv.org/abs/2109.06349 · HTML: https://mlsanity.com/paper/2109.06349/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [Semantic Parsing for Task Oriented Dialog using Hierarchical Representations](https://mlsanity.com/paper/1810.07942/) (2018)
- [DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue](https://mlsanity.com/paper/2009.13570/) (2020)
- [Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders](https://mlsanity.com/paper/2104.08027/) (2021)
- [Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking](https://mlsanity.com/paper/1910.03544/) (2019)
- [Contrastive Self-supervised Sequential Recommendation with Robust Augmentation](https://mlsanity.com/paper/2108.06479/) (2021)
- [Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference](https://mlsanity.com/paper/2010.13009/) (2020)
- [Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer](https://mlsanity.com/paper/2105.00522/) (2021)
- [Incremental Few-shot Text Classification with Multi-round New Classes: Formulation, Dataset and System](https://mlsanity.com/paper/2104.11882/) (2021)
- [Example-Driven Intent Prediction with Observers](https://mlsanity.com/paper/2010.08684/) (2020)
- [CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detection](https://mlsanity.com/paper/2004.01881/) (2020)
- [Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detection](https://mlsanity.com/paper/2106.04564/) (2021)
- [Composed Variational Natural Language Generation for Few-shot Intents](https://mlsanity.com/paper/2009.10056/) (2020)
- [Basket Recommendation with Multi-Intent Translation Graph Neural Network](https://mlsanity.com/paper/2010.11419/) (2020)
- [Pseudo Siamese Network for Few-shot Intent Generation](https://mlsanity.com/paper/2105.00896/) (2021)

## Built on by (most cited first, 30 of 46)

- [Improving In-Context Few-Shot Learning via Self-Supervised Training](https://mlsanity.com/paper/2205.01703/) (2022)
- [ConDA: Contrastive Domain Adaptation for AI-generated Text Detection](https://mlsanity.com/paper/2309.03992/) (2023)
- [Generalized Category Discovery with Decoupled Prototypical Network](https://mlsanity.com/paper/2211.15115/) (2022)
- [SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation](https://mlsanity.com/paper/2209.06664/) (2022)
- [New Intent Discovery with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2205.12914/) (2022)
- [SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding](https://mlsanity.com/paper/2209.06638/) (2022)
- [Mask-guided BERT for Few Shot Text Classification](https://mlsanity.com/paper/2302.10447/) (2023)
- [PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation](https://mlsanity.com/paper/2310.14192/) (2023)
- [Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information](https://mlsanity.com/paper/2302.05096/) (2023)
- [Exploring Zero and Few-shot Techniques for Intent Classification](https://mlsanity.com/paper/2305.07157/) (2023)
- [Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization](https://mlsanity.com/paper/2205.07208/) (2022)
- [Whodunit? Learning to Contrast for Authorship Attribution](https://mlsanity.com/paper/2209.11887/) (2022)
- [Is Contrasting All You Need? Contrastive Learning for the Detection and Attribution of AI-generated Text](https://mlsanity.com/paper/2407.09364/) (2024)
- [Improving Fine-tuning of Self-supervised Models with Contrastive Initialization](https://mlsanity.com/paper/2208.00238/) (2022)
- [NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue](https://mlsanity.com/paper/2204.13021/) (2022)
- [IntentGPT: Few-shot Intent Discovery with Large Language Models](https://mlsanity.com/paper/2411.10670/) (2024)
- [Emergent Quantized Communication](https://mlsanity.com/paper/2211.02412/) (2022)
- [Hallucination Detection for Grounded Instruction Generation](https://mlsanity.com/paper/2310.15319/) (2023)
- [Transfer and Alignment Network for Generalized Category Discovery](https://mlsanity.com/paper/2312.16467/) (2023)
- [New Intent Discovery with Attracting and Dispersing Prototype](https://mlsanity.com/paper/2403.16913/) (2024)
- [Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classification](https://mlsanity.com/paper/2412.15603/) (2024)
- [QAID: Question Answering Inspired Few-shot Intent Detection](https://mlsanity.com/paper/2303.01593/) (2023)
- [Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-training](https://mlsanity.com/paper/2306.05278/) (2023)
- [InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2310.08885/) (2023)
- [ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Filler](https://mlsanity.com/paper/2403.17536/) (2024)
- [Learning to Select from Multiple Options](https://mlsanity.com/paper/2212.00301/) (2022)
- [Domain Adaptation in Intent Classification Systems: A Review](https://mlsanity.com/paper/2404.14415/) (2024)
- [Learning Better Intent Representations for Financial Open Intent Classification](https://mlsanity.com/paper/2210.14304/) (2022)
- [Dense Retrieval as Indirect Supervision for Large-space Decision Making](https://mlsanity.com/paper/2310.18619/) (2023)
- [The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classification](https://mlsanity.com/paper/2210.06440/) (2022)

## Abstract (arXiv)

In this work, we focus on a more challenging few-shot intent detection scenario where many intents are fine-grained and semantically similar. We present a simple yet effective few-shot intent detection schema via contrastive pre-training and fine-tuning. Specifically, we first conduct self-supervised contrastive pre-training on collected intent datasets, which implicitly learns to discriminate semantically similar utterances without using any labels. We then perform few-shot intent detection together with supervised contrastive learning, which explicitly pulls utterances from the same intent closer and pushes utterances across different intents farther. Experimental results show that our proposed method achieves state-of-the-art performance on three challenging intent detection datasets under 5-shot and 10-shot settings.