mlsanityData to 14 Oct 2026

Minimizing PLM-Based Few-Shot Intent Detectors

arXiv 2407.09943 ↗First posted 13 Jul 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent research has demonstrated the feasibility of training efficient intent detectors based on pre-trained language model~(PLM) with limited labeled data. However, deploying these detectors in resource-constrained environments such as mobile devices poses challenges due to their large sizes. In this work, we aim to address this issue by exploring techniques to minimize the size of PLM-based intent detectors trained with few-shot data. Specifically, we utilize large language models (LLMs) for data augmentation, employ a cutting-edge model compression method for knowledge distillation, and devise a vocabulary pruning mechanism called V-Prune. Through these approaches, we successfully achieve a compression ratio of 21 in model memory usage, including both Transformer and the vocabulary, while maintaining almost identical performance levels on four real-world benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 18 references that are in the map. First posted on arXiv 13 Jul 2024.

Builds on

11 of 18 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  4. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  5. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  6. 2021Effectiveness of Pre-training for Few-shot Intent Classificationcited by 27
  7. 2020HINT3: Raising the bar for Intent Detection in the Wildcited by 21
  8. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  9. 2022Knowledge Distillation of Russian Language Models with Reduction of Vocabularycited by 7
  10. 2020Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settingscited by 6
  11. 2023Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-trainingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  2. 2024A Survey of Small Language Models
  3. 2023Just CHOP: Embarrassingly Simple LLM Compression
  4. 2024Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
  5. 2024Aggressive Post-Training Compression on Extremely Large Language Models
  6. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  7. 2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models
  8. 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning