# How fine can fine-tuning be? Learning efficient language models

arXiv: https://arxiv.org/abs/2004.14129 · HTML: https://mlsanity.com/paper/2004.14129/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2020.

## Built on by (most cited first, 30 of 41)

- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)
- [LLMs Will Always Hallucinate, and We Need to Live With This](https://mlsanity.com/paper/2409.05746/) (2024)
- [Parameter-Efficient Fine-Tuning Design Spaces](https://mlsanity.com/paper/2301.01821/) (2023)
- [Low-Complexity Probing via Finding Subnetworks](https://mlsanity.com/paper/2104.03514/) (2021)
- [Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis](https://mlsanity.com/paper/2310.10477/) (2023)
- [Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning](https://mlsanity.com/paper/2504.02922/) (2025)
- [Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators](https://mlsanity.com/paper/2106.02205/) (2021)
- [SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models](https://mlsanity.com/paper/2303.10464/) (2023)
- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [Control Prefixes for Parameter-Efficient Text Generation](https://mlsanity.com/paper/2110.08329/) (2021)
- [How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy](https://mlsanity.com/paper/2109.04740/) (2021)
- [HFT: Half Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2404.18466/) (2024)
- [Less is More: Selective Layer Finetuning with SubTuning](https://mlsanity.com/paper/2302.06354/) (2023)
- [Towards Structured Dynamic Sparse Pre-Training of BERT](https://mlsanity.com/paper/2108.06277/) (2021)
- [Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training](https://mlsanity.com/paper/2204.11218/) (2022)
- [Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model](https://mlsanity.com/paper/2205.11833/) (2022)
- [A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models](https://mlsanity.com/paper/2210.05211/) (2022)
- [Flexible and Adaptable Summarization via Expertise Separation](https://mlsanity.com/paper/2406.05360/) (2024)
- [Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning](https://mlsanity.com/paper/2110.14943/) (2021)
- [LayerNorm: A key component in parameter-efficient fine-tuning](https://mlsanity.com/paper/2403.20284/) (2024)
- [Revolutionizing Bridge Operation and Maintenance with LLM-based Agents: An Overview of Applications and Insights](https://mlsanity.com/paper/2407.10064/) (2024)
- [HyperMLP: An Integrated Perspective for Sequence Modeling](https://mlsanity.com/paper/2602.12601/) (2026)
- [Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense](https://mlsanity.com/paper/2109.02572/) (2021)
- [Two-Stage Fine-Tuning: A Novel Strategy for Learning Class-Imbalanced Data](https://mlsanity.com/paper/2207.10858/) (2022)
- [Decom-Renorm-Merge: Model Merging on the Right Space Improves Multitasking](https://mlsanity.com/paper/2505.23117/) (2025)
- [Mitigating Gradient Inversion Risks in Language Models via Token Obfuscation](https://mlsanity.com/paper/2602.15897/) (2026)
- [Continual Learning with Global Alignment](https://mlsanity.com/paper/2205.12186/) (2022)
- [A Stability Analysis of Fine-Tuning a Pre-Trained Model](https://mlsanity.com/paper/2301.09820/) (2023)

## Abstract (arXiv)

State-of-the-art performance on language understanding tasks is now achieved with increasingly large networks; the current record holder has billions of parameters. Given a language model pre-trained on massive unlabeled text corpora, only very light supervised fine-tuning is needed to learn a task: the number of fine-tuning steps is typically five orders of magnitude lower than the total parameter count. Does this mean that fine-tuning only introduces small differences from the pre-trained model in the parameter space? If so, can one avoid storing and computing an entire model for each task? In this work, we address these questions by using Bidirectional Encoder Representations from Transformers (BERT) as an example. As expected, we find that the fine-tuned models are close in parameter space to the pre-trained one, with the closeness varying from layer to layer. We show that it suffices to fine-tune only the most critical layers. Further, we find that there are surprisingly many good solutions in the set of sparsified versions of the pre-trained model. As a result, fine-tuning of huge language models can be achieved by simply setting a certain number of entries in certain layers of the pre-trained parameters to zero, saving both task-specific parameter storage and computational cost.