mlsanityData to 7 Oct 2026

Effectiveness of Pre-training for Few-shot Intent Classification

arXiv 2109.05782 ↗First posted 13 Sep 2021cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

This paper investigates the effectiveness of pre-training for few-shot intent classification. While existing paradigms commonly further pre-train language models such as BERT on a vast amount of unlabeled corpus, we find it highly effective and efficient to simply fine-tune BERT with a small set of labeled utterances from public datasets. Specifically, fine-tuning BERT with roughly 1,000 labeled data yields a pre-trained model -- IntentBERT, which can easily surpass the performance of existing pre-trained models for few-shot intent classification on novel domains with very different semantics. The high effectiveness of IntentBERT confirms the feasibility and practicality of few-shot intent detection, and its high generalization ability across different domains suggests that intent classification tasks may share a similar underlying structure, which can be efficiently learned from a small set of labeled data. The source code can be found at https://github.com/hdzhang-code/IntentBERT.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

23 of 30 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  9. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  10. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  11. 2020TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialoguecited by 185
  12. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
11 more
  1. 2017A Survey on Dialogue Systems: Recent Advances and New Frontierscited by 162
  2. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  3. 2020DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialoguecited by 89
  4. 2020Recent Advances and Challenges in Task-oriented Dialog Systemcited by 78
  5. 2020SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teachingcited by 71
  6. 2019Induction Networks for Few-Shot Text Classificationcited by 64
  7. 2020Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inferencecited by 47
  8. 2020HINT3: Raising the bar for Intent Detection in the Wildcited by 21
  9. 2020Intent Detection with WikiHowcited by 19
  10. 2020Composed Variational Natural Language Generation for Few-shot Intentscited by 19
  11. 2020Dynamic Semantic Matching and Aggregation Network for Few-shot Intent Detectioncited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 11; 2024: 5; 2025: 4; 2026: 1.

27 papers.

Sort:

By area:

  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  2. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 35
  3. 2023Towards A Holistic Landscape of Situated Theory of Mind in Large Language Modelscited by 32
  4. 2022New Intent Discovery with Pre-training and Contrastive Learningcited by 28
  5. 2024Intent Detection in the Age of LLMscited by 25
  6. 2023Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Informationcited by 24
  7. 2023IDAS: Intent Discovery with Abstractive Summarizationcited by 22
  8. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  9. 2022Improving the Sample Efficiency of Prompt Tuning with Domain Adaptationcited by 15
  10. 2023Intent Induction from Conversations for Task-Oriented Dialogue Track at DSTC 11cited by 10
  11. 2022NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialoguecited by 8
  12. 2022Less is More: Parameter-Free Text Classification with Gzipcited by 7
15 more
  1. 2023Bringing the State-of-the-Art to Customers: A Neural Agent Assistant Framework for Customer Service Supportcited by 6
  2. 2023QAID: Question Answering Inspired Few-shot Intent Detectioncited by 5
  3. 2023Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-trainingcited by 5
  4. 2025Delegated Authorization for Agents Constrained to Semantic Task-to-Scope Matchingcited by 5
  5. 2025LANID: LLM-assisted New Intent Discoverycited by 3
  6. 2024Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoderscited by 2
  7. 2023Multi-Tenant Optimization For Few-Shot Task-Oriented FAQ Retrievalcited by 1
  8. 2023RSVP: Customer Intent Detection via Agent Response Contrastive and Generative Pre-Trainingcited by 1
  9. 2024Practical token pruning for foundation models in few-shot conversational virtual assistant systemscited by 1
  10. 2024Generating Synthetic Datasets for Few-shot Prompt Tuningcited by 1
  11. 2025Few-Shot Query Intent Detection via Relation-Aware Prompt Learningcited by 1
  12. 2023Multi-View Zero-Shot Open Intent Induction from Dialogues: Multi Domain Batch and Proxy Gradient Transfercited by 0
  13. 2024Minimizing PLM-Based Few-Shot Intent Detectorscited by 0
  14. 2025Building a Few-Shot Cross-Domain Multilingual NLU Model for Customer Carecited by 0
  15. 2026Hybrid Inspection and Task-Based Access Control in Zero-Trust Agentic AIcited by 0