mlsanityData to 7 Oct 2026

Parameter-Efficient Tuning Makes a Good Classification Head

arXiv 2210.16771 ↗First posted 30 Oct 2022cs.CL cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

In recent years, pretrained models revolutionized the paradigm of natural language understanding (NLU), where we append a randomly initialized classification head after the pretrained backbone, e.g. BERT, and finetune the whole model. As the pretrained backbone makes a major contribution to the improvement, we naturally expect a good pretrained classification head can also benefit the training. However, the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning, making the usual head-only pretraining (LP-FT) ineffective. In this paper, we find that parameter-efficient tuning makes a good classification head, with which we can simply replace the randomly initialized heads for a stable performance gain. Our experiments demonstrate that the classification head jointly pretrained with parameter-efficient tuning consistently improves the performance on 9 tasks in GLUE and SuperGLUE.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 50 of its 65 references that are in the map. First posted on arXiv 30 Oct 2022.

Builds on

50 of 65 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2014How transferable are features in deep neural networks?cited by 1,216
  12. 2019Attention is not Explanationcited by 916
  13. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  14. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  15. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  16. 2021GPT Understands, Toocited by 715
  17. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  18. 2019Attention is not not Explanationcited by 629
  19. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  20. 2021Editing Factual Knowledge in Language Modelscited by 548
  21. 2019Is Attention Interpretable?cited by 387
  22. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  23. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  24. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  25. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  26. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  27. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  28. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  29. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  30. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  31. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  32. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  33. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  34. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  35. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  36. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  37. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  38. 2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 8; 2025: 2; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systemscited by 20
  2. 2024Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learningcited by 19
  3. 2024RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarizationcited by 10
  4. 2023FUN with Fisher: Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezingcited by 8
  5. 2024FourierKAN outperforms MLP on Text Classification Head Fine-tuningcited by 5
  6. 2023Test-Time Training for Speechcited by 4
  7. 2023MT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extractioncited by 3
  8. 2024BiSSL: Enhancing the Alignment Between Self-Supervised Pretraining and Downstream Fine-Tuning via Bilevel Optimizationcited by 3
  9. 2024Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learningcited by 3
  10. 2023PVP: Pre-trained Visual Parameter-Efficient Tuningcited by 1
  11. 2024Decouple then Converge: Handling Unknown Unlabeled Distributions in Long-Tailed Semi-Supervised Learningcited by 1
  12. 2025Are LLMs Enough for Hyperpartisan, Fake, Polarized and Harmful Content Detection? Evaluating In-Context Learning vs. Fine-Tuningcited by 1
3 more
  1. 2026A Theoretical Framework for LLM Fine-tuning Using Early Stopping for Non-random Initializationcited by 1
  2. 2024Exploring prompts to elicit memorization in masked language model-based named entity recognitioncited by 0
  3. 2025Can Smaller LLMs do better? Unlocking Cross-Domain Potential through Parameter-Efficient Fine-Tuning for Text Summarizationcited by 0