# STAR: Spectral Truncation and Rescale for Model Merging

arXiv: https://arxiv.org/abs/2502.10339 · HTML: https://mlsanity.com/paper/2502.10339/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 18 of 18)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts](https://mlsanity.com/paper/1307.5336/) (2013)
- [Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities](https://mlsanity.com/paper/2408.07666/) (2024)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks](https://mlsanity.com/paper/2312.06795/) (2023)
- [Localizing Task Information for Improved Model Merging and Compression](https://mlsanity.com/paper/2405.07813/) (2024)
- [EMR-Merging: Tuning-Free High-Performance Model Merging](https://mlsanity.com/paper/2405.17461/) (2024)
- [Model Merging by Uncertainty-Based Gradient Matching](https://mlsanity.com/paper/2310.12808/) (2023)
- [Re-basin via implicit Sinkhorn differentiation](https://mlsanity.com/paper/2212.12042/) (2022)
- [MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic](https://mlsanity.com/paper/2406.11385/) (2024)
- [Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead](https://mlsanity.com/paper/2407.00066/) (2024)

## Built on by (most cited first, 11 of 11)

- [AdaRank: Adaptive Rank Pruning for Enhanced Model Merging](https://mlsanity.com/paper/2503.22178/) (2025)
- [Why Do More Experts Fail? A Theoretical Analysis of Model Merging](https://mlsanity.com/paper/2505.21226/) (2025)
- [When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging](https://mlsanity.com/paper/2602.05536/) (2026)
- [Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Merging](https://mlsanity.com/paper/2606.01717/) (2026)
- [Locket: Robust Feature-Locking Technique for Language Models](https://mlsanity.com/paper/2510.12117/) (2025)
- [ResMerge: Residual-based Spectral Merging of Large Language Models](https://mlsanity.com/paper/2606.02252/) (2026)
- [Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis](https://mlsanity.com/paper/2609.00746/) (2026)
- [From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models](https://mlsanity.com/paper/2609.19553/) (2026)
- [Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks](https://mlsanity.com/paper/2604.12935/) (2026)
- [Training-Free Task Vectors for LLM Behavioral Control](https://mlsanity.com/paper/2609.09054/) (2026)
- [Orthogonal Yet Coupled: Decoupling Geometric Components for Model Merging](https://mlsanity.com/paper/2609.37564/) (2026)

## Abstract (arXiv)

Model merging is an efficient way of obtaining a multi-task model from several pretrained models without further fine-tuning, and it has gained attention in various domains, including natural language processing (NLP). Despite the efficiency, a key challenge in model merging is the seemingly inevitable decrease in task performance as the number of models increases. In this paper, we propose Spectral Truncation And Rescale (STAR) that aims at mitigating ``merging conflicts'' by truncating small components in the respective spectral spaces, which is followed by an automatic parameter rescaling scheme to retain the nuclear norm of the original matrix. STAR requires no additional inference on original training data and is robust to hyperparamater choice. We demonstrate the effectiveness of STAR through extensive model merging cases on diverse NLP tasks. Specifically, STAR works robustly across varying model sizes, and can outperform baselines by 4.2% when merging 12 models on Flan-T5. Our code is publicly available at https://github.com/IBM/STAR.