mlsanityData to 14 Oct 2026

Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Tasks

arXiv 2210.12022 ↗First posted 21 Oct 2022cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (LMs) obtain state-of-the-art performance when adapted to text classification tasks. However, when using such models in real-world applications, efficiency considerations are paramount. In this paper, we study how different training procedures that adapt LMs to text classification perform, as we vary model and train set size. More specifically, we compare standard fine-tuning, prompting, and knowledge distillation (KD) when the teacher was trained with either fine-tuning or prompting. Our findings suggest that even though fine-tuning and prompting work well to train large LMs on large train sets, there are more efficient alternatives that can reduce compute or data cost. Interestingly, we find that prompting combined with KD can reduce compute and data cost at the same time.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 25 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

20 of 25 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  10. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  12. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
8 more
  1. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  2. 2021How Many Data Points is a Prompt Worth?cited by 198
  3. 2021Generating Datasets with Pretrained Language Modelscited by 193
  4. 2021The Efficiency Misnomercited by 98
  5. 2021How to Train BERT with an Academic Budgetcited by 80
  6. 2021RAFT: A Real-World Few-Shot Text Classification Benchmarkcited by 44
  7. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
  8. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale
  2. 2024Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance
  3. 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective
  4. 2024PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
  5. 2025Large Language Models For Text Classification: Case Study And Comprehensive Review
  6. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Models
  7. 2022Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes