# Linear Interpolation In Parameter Space is Good Enough for Fine-Tuned Language Models

arXiv: https://arxiv.org/abs/2211.12092 · HTML: https://mlsanity.com/paper/2211.12092/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 22 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [SemEval-2017 Task 4: Sentiment Analysis in Twitter](https://mlsanity.com/paper/1912.00741/) (2019)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes](https://mlsanity.com/paper/2104.11044/) (2021)

## Built on by (most cited first, 7 of 7)

- [Learn Your Reference Model for Real Good Alignment](https://mlsanity.com/paper/2404.09656/) (2024)
- [WARP: On the Benefits of Weight Averaged Rewarded Policies](https://mlsanity.com/paper/2406.16768/) (2024)
- [How to Merge Your Multimodal Models Over Time?](https://mlsanity.com/paper/2412.06712/) (2024)
- [Subspace-Boosted Model Merging](https://mlsanity.com/paper/2506.16506/) (2025)
- [Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning](https://mlsanity.com/paper/2304.10309/) (2023)
- [Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging](https://mlsanity.com/paper/2410.12937/) (2024)
- [AsyncSwitch: Asynchronous Text-Speech Adaptation for Code-Switched ASR](https://mlsanity.com/paper/2506.14190/) (2025)

## Abstract (arXiv)

The simplest way to obtain continuous interpolation between two points in high dimensional space is to draw a line between them. While previous works focused on the general connectivity between model parameters, we explored linear interpolation for parameters of pre-trained models after fine-tuning. Surprisingly, we could perform linear interpolation without a performance drop in intermediate points for fine-tuned models. For controllable text generation, such interpolation could be seen as moving a model towards or against the desired text attribute (e.g., positive sentiment), which could be used as grounds for further methods for controllable text generation without inference speed overhead.