mlsanityData to 7 Oct 2026

Label Semantic Aware Pre-training for Few-shot Text Classification

arXiv 2204.07128 ↗First posted 14 Apr 2022cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

In text classification tasks, useful information is encoded in the label names. Label semantic aware systems have leveraged this information for improved text classification performance during fine-tuning and prediction. However, use of label-semantics during pre-training has not been extensively explored. We therefore propose Label Semantic Aware Pre-training (LSAP) to improve the generalization and data efficiency of text classification systems. LSAP incorporates label semantics into pre-trained generative models (T5 in our case) by performing secondary pre-training on labeled sentences from a variety of domains. As domain-general pre-training requires large amounts of data, we develop a filtering and labeling pipeline to automatically create sentence-label pairs from unlabeled text. We perform experiments on intent (ATIS, Snips, TOPv2) and topic classification (AG News, Yahoo! Answers). LSAP obtains significant accuracy improvements over state-of-the-art models for few-shot text classification while maintaining performance comparable to state of the art in high-resource settings.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

37 of 50 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
25 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  2. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  3. 2015Pointer Networkscited by 882
  4. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  5. 2018Universal Sentence Encodercited by 485
  6. 2019Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Datasetcited by 426
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  8. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  9. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 231
  10. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  11. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  12. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  13. 2020Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsingcited by 72
  14. 2020Don't Parse, Generate! A Sequence to Sequence Architecture for Task-Oriented Semantic Parsingcited by 59
  15. 2020Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inferencecited by 47
  16. 2020An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labelscited by 44
  17. 2020Span-ConveRT: Few-shot Span Extraction for Dialog with Pretrained Conversational Representationscited by 42
  18. 2021Few-shot Intent Classification and Slot Filling with Retrieved Examplescited by 35
  19. 2020Learning to Classify Intents and Slot Labels Given a Handful of Examplescited by 31
  20. 2020Augmented Natural Language for Generative Sequence Labelingcited by 29
  21. 2020Dynamic Memory Induction Networks for Few-Shot Text Classificationcited by 23
  22. 2022Label Semantics for Few Shot Named Entity Recognitioncited by 21
  23. 2020Intent Detection with WikiHowcited by 19
  24. 2020Data-Efficient Pretraining via Contrastive Self-Supervisioncited by 14
  25. 2020Modeling Label Semantics for Predicting Emotional Reactionscited by 6

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 10; 2024: 3; 2025: 2.

18 papers.

Sort:

By area:

  1. 2023Mask-guided BERT for Few Shot Text Classificationcited by 25
  2. 2022Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspectivecited by 13
  3. 2023Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineeringcited by 13
  4. 2023UniEX: An Effective and Efficient Framework for Unified Information Extraction via a Span-extractive Perspectivecited by 6
  5. 2024HiGen: Hierarchy-Aware Sequence Generation for Hierarchical Text Classificationcited by 6
  6. 2023Pre-training Intent-Aware Encoders for Zero- and Few-Shot Intent Classificationcited by 5
  7. 2022Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Fieldcited by 4
  8. 2023Mimicking the Thinking Process for Emotion Recognition in Conversation with Prompts and Paraphrasingcited by 4
  9. 2022The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classificationcited by 2
  10. 2023Description-Enhanced Label Embedding Contrastive Learning for Text Classificationcited by 2
  11. 2023Meta-training with Demonstration Retrieval for Efficient Few-shot Learningcited by 2
  12. 2024Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoderscited by 2
6 more
  1. 2025InstructNet: A Novel Approach for Multi-Label Instruction Classification through Advanced Deep Learningcited by 2
  2. 2023Low-Resource Compositional Semantic Parsing with Concept Pretrainingcited by 1
  3. 2023All Labels Together: Low-shot Intent Detection with an Efficient Label Semantic Encoding Paradigmcited by 1
  4. 2025Semantic Anchors in In-Context Learning: Why Small LLMs Cannot Flip Their Labelscited by 1
  5. 2023NER-to-MRC: Named-Entity Recognition Completely Solving as Machine Reading Comprehensioncited by 0
  6. 2024Exploring Description-Augmented Dataless Intent Classificationcited by 0