# Parameter-Efficient Tuning on Layer Normalization for Pre-trained Language Models

arXiv: https://arxiv.org/abs/2211.08682 · HTML: https://mlsanity.com/paper/2211.08682/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Parameter-Efficient Transfer Learning with Diff Pruning](https://mlsanity.com/paper/2012.07463/) (2020)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning](https://mlsanity.com/paper/2110.07577/) (2021)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [DART: Open-Domain Structured Data Record to Text Generation](https://mlsanity.com/paper/2007.02871/) (2020)
- [Robust Transfer Learning with Pretrained Language Models through Adapters](https://mlsanity.com/paper/2108.02340/) (2021)
- [Encoder-Agnostic Adaptation for Conditional Language Generation](https://mlsanity.com/paper/1908.06938/) (2019)
- [Parameter-Efficient Tuning by Manipulating Hidden States of Pretrained Language Models For Classification Tasks](https://mlsanity.com/paper/2204.04596/) (2022)

## Built on by (most cited first, 19 of 19)

- [Deepfake Detection that Generalizes Across Benchmarks](https://mlsanity.com/paper/2508.06248/) (2025)
- [Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models](https://mlsanity.com/paper/2407.12616/) (2024)
- [Adapt Your Teacher: Improving Knowledge Distillation for Exemplar-free Continual Learning](https://mlsanity.com/paper/2308.09544/) (2023)
- [Evaluating Parameter Efficient Methods for RLVR](https://mlsanity.com/paper/2512.23165/) (2025)
- [Unlocking the Hidden Potential of CLIP in Generalizable Deepfake Detection](https://mlsanity.com/paper/2503.19683/) (2025)
- [MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling](https://mlsanity.com/paper/2503.11144/) (2025)
- [PEFT-Ref: A Modular Reference Architecture and Typology for Parameter-Efficient Finetuning Techniques](https://mlsanity.com/paper/2304.12410/) (2023)
- [Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe](https://mlsanity.com/paper/2406.04165/) (2024)
- [UP-Person: Unified Parameter-Efficient Transfer Learning for Text-based Person Retrieval](https://mlsanity.com/paper/2504.10084/) (2025)
- [Forecast-PEFT: Parameter-Efficient Fine-Tuning for Pre-trained Motion Forecasting Models](https://mlsanity.com/paper/2407.19564/) (2024)
- [iConFormer: Dynamic Parameter-Efficient Tuning with Input-Conditioned Adaptation](https://mlsanity.com/paper/2409.02838/) (2024)
- [Exploiting Layer Normalization Fine-tuning in Visual Transformer Foundation Models for Classification](https://mlsanity.com/paper/2508.07577/) (2025)
- [MetaWeather: Few-Shot Weather-Degraded Image Restoration](https://mlsanity.com/paper/2308.14334/) (2023)
- [CLAMP: Contrastive LAnguage Model Prompt-tuning](https://mlsanity.com/paper/2312.01629/) (2023)
- [Context-PEFT: Efficient Multi-Modal, Multi-Task Fine-Tuning](https://mlsanity.com/paper/2312.08900/) (2023)
- [Deep Generative Continual Learning using Functional LoRA: FunLoRA](https://mlsanity.com/paper/2510.02631/) (2025)
- [The Alpha Blending Hypothesis: Compositing Shortcut in Deepfake Detection](https://mlsanity.com/paper/2605.10334/) (2026)
- [Robust Promptable Video Object Segmentation](https://mlsanity.com/paper/2605.12006/) (2026)
- [FedLNS: Leverage LayerNorm Signature Modeling to Mitigate Adversarial Manipulation in Federated LLMs](https://mlsanity.com/paper/2608.18736/) (2026)

## Abstract (arXiv)

Conventional fine-tuning encounters increasing difficulties given the size of current Pre-trained Language Models, which makes parameter-efficient tuning become the focal point of frontier research. Previous methods in this field add tunable adapters into MHA or/and FFN of Transformer blocks to enable PLMs achieve transferability. However, as an important part of Transformer architecture, the power of layer normalization for parameter-efficent tuning is ignored. In this paper, we first propose LN-tuning, by tuning the gain and bias term of Layer Normalization module with only 0.03% parameters, which is of high time-efficency and significantly superior to baselines which are less than 0.1% tunable parameters. Further, we study the unified framework of combining LN-tuning with previous ones and we find that: (1) the unified framework of combining prefix-tuning, the adapter-based method working on MHA, and LN-tuning achieves SOTA performance. (2) unified framework which tunes MHA and LayerNorm simultaneously can get performance improvement but those which tune FFN and LayerNorm simultaneous will cause performance decrease. Ablation study validates LN-tuning is of no abundant parameters and gives a further understanding of it.