mlsanityData to 7 Oct 2026

LayerNorm: A key component in parameter-efficient fine-tuning

arXiv 2403.20284 ↗First posted 29 Mar 2024cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Fine-tuning a pre-trained model, such as Bidirectional Encoder Representations from Transformers (BERT), has been proven to be an effective method for solving many natural language processing (NLP) tasks. However, due to the large number of parameters in many state-of-the-art NLP models, including BERT, the process of fine-tuning is computationally expensive. One attractive solution to this issue is parameter-efficient fine-tuning, which involves modifying only a minimal segment of the model while keeping the remainder unchanged. Yet, it remains unclear which segment of the BERT model is crucial for fine-tuning. In this paper, we first analyze different components in the BERT model to pinpoint which one undergoes the most significant changes after fine-tuning. We find that output LayerNorm changes more than any other components when fine-tuned for different General Language Understanding Evaluation (GLUE) tasks. Then we show that only fine-tuning the LayerNorm can reach comparable, or in some cases better, performance to full fine-tuning and other parameter-efficient fine-tuning methods. Moreover, we use Fisher information to determine the most critical subset of LayerNorm and demonstrate that many NLP tasks in the GLUE benchmark can be solved by fine-tuning only a small portion of LayerNorm with negligible performance degradation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 44 of its 66 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

44 of 66 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
32 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  5. 2018Neural Network Acceptability Judgmentscited by 952
  6. 2020On Layer Normalization in the Transformer Architecturecited by 906
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  9. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  10. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  11. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  12. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  13. 2019Understanding and Improving Layer Normalizationcited by 213
  14. 2016Deep Information Propagationcited by 194
  15. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  16. 2021The Principles of Deep Learning Theorycited by 169
  17. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  18. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  19. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  20. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
  21. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  22. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  23. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  24. 2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 87
  25. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  26. 2023Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterizationcited by 64
  27. 2019A Mean Field Theory of Batch Normalizationcited by 58
  28. 2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
  29. 2020How fine can fine-tuning be? Learning efficient language modelscited by 41
  30. 2016Reducing the Model Order of Deep Neural Networks Using Information Theorycited by 10
  31. 2020Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Predictioncited by 2
  32. 2022Fine-Tuning BERT for Automatic ADME Semantic Labeling in FDA Drug Labeling to Enhance Product-Specific Guidance Assessmentcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Deepfake Detection that Generalizes Across Benchmarkscited by 20
  2. 2024Learning Modality-Aware Representations: Adaptive Group-wise Interaction Network for Multimodal MRI Synthesiscited by 4
  3. 2025Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classificationcited by 3
  4. 2026Robust Promptable Video Object Segmentationcited by 0