# Weighted-Reward Preference Optimization for Implicit Model Fusion

arXiv: https://arxiv.org/abs/2412.03187 · HTML: https://mlsanity.com/paper/2412.03187/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 1 of its 60 references that are in the map.
- First posted on arXiv 4 Dec 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 1 of 1)

- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)

## Built on by (most cited first, 15 of 15)

- [SEA-LION: Southeast Asian Languages in One Network](https://mlsanity.com/paper/2504.05747/) (2025)
- [MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge](https://mlsanity.com/paper/2507.21183/) (2025)
- [InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models](https://mlsanity.com/paper/2505.13878/) (2025)
- [Knowledge Fusion of Large Language Models Via Modular SkillPacks](https://mlsanity.com/paper/2505.18502/) (2025)
- [CortexDebate: Debating Sparsely and Equally for Multi-Agent Debate](https://mlsanity.com/paper/2507.03928/) (2025)
- [FuseChat-3.0: Preference Optimization Meets Heterogeneous Model Fusion](https://mlsanity.com/paper/2503.04222/) (2025)
- [Entropy-Based Adaptive Weighting for Self-Training](https://mlsanity.com/paper/2503.23913/) (2025)
- [Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration](https://mlsanity.com/paper/2506.15721/) (2025)
- [FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion](https://mlsanity.com/paper/2504.06562/) (2025)
- [Skill Weaving: Efficient LLM Improvement via Modular Skillpacks](https://mlsanity.com/paper/2605.22205/) (2026)
- [Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines](https://mlsanity.com/paper/2605.26442/) (2026)
- [An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems](https://mlsanity.com/paper/2501.00562/) (2024)
- [DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)](https://mlsanity.com/paper/2507.07855/) (2025)
- [ATLAS: A Multi-LLM Training Framework for EvoDPO with Adaptive Reference Evolution](https://mlsanity.com/paper/2602.02709/) (2026)
- [CoMerge: Conflict-Driven Preference Optimization for Multi-Task Model Merging](https://mlsanity.com/paper/2609.02273/) (2026)

## Abstract (arXiv)

While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against GPT-4-0314 on Arena-Hard. Our code is available at https://github.com/SLIT-AI/WRPO.