# Sparsity Curse: Understanding RLVR Model Parameter Space from Model Merging

arXiv: https://arxiv.org/abs/2606.18521 · HTML: https://mlsanity.com/paper/2606.18521/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 51 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Builds on (most cited first, 30 of 32)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Model merging with SVD to tie the Knots](https://mlsanity.com/paper/2410.19735/) (2024)
- [WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models](https://mlsanity.com/paper/2405.14768/) (2024)
- [On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting](https://mlsanity.com/paper/2508.11408/) (2025)
- [GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents](https://mlsanity.com/paper/2505.15810/) (2025)
- [Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering](https://mlsanity.com/paper/2409.16167/) (2024)
- [Reinforcement Learning Finetunes Small Subnetworks in Large Language Models](https://mlsanity.com/paper/2505.11711/) (2025)
- [RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness](https://mlsanity.com/paper/2502.17159/) (2025)
- [Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging](https://mlsanity.com/paper/2505.22934/) (2025)
- [RL Is Neither a Panacea Nor a Mirage: Understanding Supervised vs. Reinforcement Learning Fine-Tuning for LLMs](https://mlsanity.com/paper/2508.16546/) (2025)
- [EvoLM: In Search of Lost Language Model Training Dynamics](https://mlsanity.com/paper/2506.16029/) (2025)
- [SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agents](https://mlsanity.com/paper/2506.21669/) (2025)
- [MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging](https://mlsanity.com/paper/2505.11883/) (2025)
- [Activation-Guided Consensus Merging for Large Language Models](https://mlsanity.com/paper/2505.14009/) (2025)
- [Behavior Knowledge Merge in Reinforced Agentic Models](https://mlsanity.com/paper/2601.13572/) (2026)
- [Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs](https://mlsanity.com/paper/2509.21044/) (2025)
- [Improving Model Fusion by Training-time Neuron Alignment with Fixed Neuron Anchors](https://mlsanity.com/paper/2402.01342/) (2024)

## Built on by (most cited first, 2 of 2)

- [Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?](https://mlsanity.com/paper/2609.32996/) (2026)
- [No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation](https://mlsanity.com/paper/2609.39405/) (2026)

## Abstract (arXiv)

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful post-training paradigm that surpasses Supervised Fine-Tuning (SFT) in eliciting reasoning intelligence and resisting catastrophic forgetting. Recent studies further reveal that RLVR induces highly sparse and off-principal parameter updates compared to SFT. This naturally raises the question: does such sparsity make RLVR models more amenable to model merging? If so, model merging would offer a scalable, training-free path to aggregate diverse reasoning capabilities from independently trained RLVR models. Surprisingly, we find the opposite, uncovering a sparsity curse: the sparse RLVR updates are spread farther apart in parameter space, forming near-orthogonal shortcuts that make aggregation inherently fragile. This is likely rooted in the stochasticity of RL optimization and the diversity of emergent reasoning patterns. Unlike SFT models that converge to shared, flat basins and merge naturally, RLVR models suffer severe degradation under standard merging methods. Through systematic empirical analysis of the update geometry, we characterize the mechanisms behind this failure and propose Sensitivity-aware Resolving Merging (SAR-Merging), a merging recipe tailored for the unique structure of RLVR parameter spaces. SAR-Merging resolves conflicts in overlapping update regions via Fisher Information-based sensitivity arbitration, followed by magnitude-aware sparsification and rescaling to preserve fragile reasoning pathways. Experiments on mathematical and coding benchmarks demonstrate that SAR-Merging substantially outperforms existing merging methods on RLVR models, enabling both single-task enhancement and multi-capability fusion.