mlsanityData to 7 Oct 2026

Practical token pruning for foundation models in few-shot conversational virtual assistant systems

arXiv 2408.11799 ↗First posted 21 Aug 2024cs.CLcited by 1

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

In an enterprise Virtual Assistant (VA) system, intent classification is the crucial component that determines how a user input is handled based on what the user wants. The VA system is expected to be a cost-efficient SaaS service with low training and inference time while achieving high accuracy even with a small number of training samples. We pretrain a transformer-based sentence embedding model with a contrastive learning objective and leverage the embedding of the model as features when training intent classification models. Our approach achieves the state-of-the-art results for few-shot scenarios and performs better than other commercial solutions on popular intent classification benchmarks. However, generating features via a transformer-based model increases the inference time, especially for longer user inputs, due to the quadratic runtime of the transformer's attention mechanism. On top of model distillation, we introduce a practical multi-task adaptation approach that configures dynamic token pruning without the need for task-specific training for intent classification. We demonstrate that this approach improves the inference speed of popular sentence transformer models without affecting model performance.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 36 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

28 of 36 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  10. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  11. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  12. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
16 more
  1. 2017Efficient Natural Language Response Suggestion for Smart Replycited by 228
  2. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  3. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  4. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  5. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  6. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  7. 2020DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialoguecited by 89
  8. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  9. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  10. 2021Effectiveness of Pre-training for Few-shot Intent Classificationcited by 27
  11. 2021Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detectioncited by 22
  12. 2023Exploring Zero and Few-shot Techniques for Intent Classificationcited by 21
  13. 2021KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillationcited by 19
  14. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  15. 2020Benchmarking Commercial Intent Detection Services with Practice-Driven Evaluationscited by 8
  16. 2023Revisit Few-shot Intent Classification with PLMs: Direct Fine-tuning vs. Continual Pre-trainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FastE: Readout-Triggered Token Compression for LLM Embedding Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Intent Detection in the Age of LLMs
  2. 2025InsTALL: Context-aware Instructional Task Assistance with Multi-modal Large Language Models
  3. 2022Zero-Shot Prompting for Implicit Intent Prediction and Recommendation with Commonsense Reasoning
  4. 2024Many Hands Make Light Work: Task-Oriented Dialogue System with Module-Based Mixture-of-Experts
  5. 2024Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
  6. 2024Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
  7. 2025The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning
  8. 2025SQAP-VLA: A Synergistic Quantization-Aware Pruning Framework for High-Performance Vision-Language-Action Models