mlsanityData to 14 Oct 2026

Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classification

arXiv 2305.14827 ↗First posted 24 May 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Intent classification (IC) plays an important role in task-oriented dialogue systems. However, IC models often generalize poorly when training without sufficient annotated examples for each user intent. We propose a novel pre-training method for text encoders that uses contrastive learning with intent psuedo-labels to produce embeddings that are well-suited for IC tasks, reducing the need for manual annotations. By applying this pre-training strategy, we also introduce Pre-trained Intent-aware Encoder (PIE), which is designed to align encodings of utterances with their intent names. Specifically, we first train a tagger to identify key phrases within utterances that are crucial for interpreting intents. We then use these extracted phrases to create examples for pre-training a text encoder in a contrastive manner. As a result, our PIE model achieves up to 5.4% and 4.0% higher accuracy than the previous state-of-the-art text encoder for the N-way zero- and one-shot settings on four IC datasets.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 20 of its 26 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

20 of 26 references.

References · 20

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  6. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 434
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  8. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  9. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 204
  10. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  11. 2018Semantic Parsing for Task Oriented Dialog using Hierarchical Representationscited by 101
  12. 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
8 more
  1. 2020Learning to Classify Intents and Slot Labels Given a Handful of Examplescited by 31
  2. 2022Label Semantic Aware Pre-training for Few-shot Text Classificationcited by 18
  3. 2023ContrastNet: A Contrastive Learning Framework for Few-Shot Text Classificationcited by 17
  4. 2021Automatic Intent-Slot Induction for Dialogue Systemscited by 13
  5. 2023Intent Induction from Conversations for Task-Oriented Dialogue Track at DSTC 11cited by 10
  6. 2021A Neural Few-Shot Text Classification Reality Checkcited by 6
  7. 2022A Simple Meta-learning Paradigm for Zero-shot Intent Classification with Mixture Attention Mechanismcited by 6
  8. 2022Template-based Approach to Zero-shot Intent Recognitioncited by 5

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 1.

5 papers.

  1. 2024Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Modelscited by 9
  2. 2024A Survey of Ontology Expansion for Conversational Understandingcited by 6
  3. 2024Dynamic Label Name Refinement for Few-Shot Dialogue Intent Classificationcited by 6
  4. 2025GLEAN: Active Generalized Category Discovery with Diverse LLM Feedbackcited by 6
  5. 2024Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoderscited by 2