mlsanityData to 7 Oct 2026

What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning

arXiv 1911.03090 ↗First posted 8 Nov 2019cs.CLcited by 87

Not on the map. Why?

Abstract (arXiv)

Pretrained transformer-based language models have achieved state of the art across countless tasks in natural language processing. These models are highly expressive, comprising at least a hundred million parameters and a dozen layers. Recent evidence suggests that only a few of the final layers need to be fine-tuned for high quality on downstream tasks. Naturally, a subsequent research question is, "how many of the last layers do we need to fine-tune?" In this paper, we precisely answer this question. We examine two recent pretrained language models, BERT and RoBERTa, across standard tasks in textual entailment, semantic similarity, sentiment analysis, and linguistic acceptability. We vary the number of final layers that are fine-tuned, then study the resulting change in task-specific effectiveness. We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality. Surprisingly, we also find that fine-tuning all layers does not always help.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 26; 2023: 15; 2024: 20; 2025: 14; 2026: 6.

87 papers.

Sort:

By area:

  1. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  2. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  3. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  4. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  5. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  6. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  7. 2022CLiMB: A Continual Learning Benchmark for Vision-and-Language Taskscited by 68
  8. 2023Structure-informed Language Models Are Protein Designerscited by 68
  9. 2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
  10. 2023Learning from models beyond fine-tuningcited by 47
  11. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  12. 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
38 more
  1. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  2. 2024Multimodal Instruction Tuning with Conditional Mixture of LoRAcited by 36
  3. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  4. 2024Continual Forgetting for Pre-trained Vision Modelscited by 29
  5. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  6. 2022Egeria: Efficient DNN Training with Knowledge-Guided Layer Freezingcited by 22
  7. 2022Acceptability Judgements via Examining the Topology of Attention Mapscited by 19
  8. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 16
  9. 2022Parameter-Efficient Tuning Makes a Good Classification Headcited by 15
  10. 2023Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptationcited by 14
  11. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasonerscited by 14
  12. 2025On Finetuning Tabular Foundation Modelscited by 13
  13. 2022HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Taskscited by 12
  14. 2022Tiny-Attention Adapter: Contexts Are More Important Than the Number of Parameterscited by 12
  15. 2024SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezingcited by 12
  16. 2022Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Trainingcited by 11
  17. 2022Representation Projection Invariance Mitigates Representation Collapsecited by 10
  18. 2022Entropy-based Stability-Plasticity for Lifelong Learningcited by 8
  19. 2023Scalable Neural Network Kernelscited by 8
  20. 2024C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learningcited by 8
  21. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 7
  22. 2022Label-enhanced Prototypical Network with Contrastive Learning for Multi-label Few-shot Aspect Category Detectioncited by 7
  23. 2024Fusion Transformer with Object Mask Guidance for Image Forgery Analysiscited by 7
  24. 2024Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Samplingcited by 7
  25. 2021Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuningcited by 6
  26. 2025Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRAcited by 6
  27. 2022Transformer-based Models of Text Normalization for Speech Applicationscited by 5
  28. 2022Embedding Recycling for Language Modelscited by 5
  29. 2023FDAPT: Federated Domain-adaptive Pre-training for Language Modelscited by 5
  30. 2023FerKD: Surgical Label Adaptation for Efficient Distillationcited by 5
  31. 2024Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuningcited by 5
  32. 2023PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniquescited by 4
  33. 2024LayerNorm: A key component in parameter-efficient fine-tuningcited by 4
  34. 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristicscited by 4
  35. 2025Practical Continual Forgetting for Pre-trained Vision Modelscited by 4
  36. 2025Memory-Efficient Fine-Tuning of Transformers via Token Selectioncited by 4
  37. 2025Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Datacited by 4
  38. 2022Evaluating Generalizability of Fine-Tuned Models for Fake News Detectioncited by 3

50 most cited of 87 All 87 (JSON)