# Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast

arXiv: https://arxiv.org/abs/2608.06400 · HTML: https://mlsanity.com/paper/2608.06400/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)
- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 19 of 19)

- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [Understanding Dataset Difficulty with V-Usable Information](https://mlsanity.com/paper/2110.08420/) (2021)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards](https://mlsanity.com/paper/2402.18571/) (2024)
- [MaxMin-RLHF: Alignment with Diverse Human Preferences](https://mlsanity.com/paper/2402.08925/) (2024)
- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [LLMs + Persona-Plug = Personalized LLMs](https://mlsanity.com/paper/2409.11901/) (2024)
- [What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data](https://mlsanity.com/paper/2510.26202/) (2025)
- [Reward Models in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2506.15421/) (2025)
- [Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences](https://mlsanity.com/paper/2405.15065/) (2024)
- [Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas](https://mlsanity.com/paper/2501.11549/) (2025)
- [Interpretable Reward Model via Sparse Autoencoder](https://mlsanity.com/paper/2508.08746/) (2025)
- [DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling](https://mlsanity.com/paper/2403.01197/) (2024)
- [Test-Time Alignment via Hypothesis Reweighting](https://mlsanity.com/paper/2412.08812/) (2024)
- [MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning](https://mlsanity.com/paper/2505.24846/) (2025)
- [Rethinking Diverse Human Preference Learning through Principal Component Analysis](https://mlsanity.com/paper/2502.13131/) (2025)
- [CoPL: Collaborative Preference Learning for Personalizing LLMs](https://mlsanity.com/paper/2503.01658/) (2025)
- [SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder](https://mlsanity.com/paper/2511.07896/) (2025)
- [Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling](https://mlsanity.com/paper/2606.04284/) (2026)

## Abstract (arXiv)

Reward models are central to learning from human preferences, yet identifying what drives their predictions remains challenging. Recent sparse Mixture-of-Experts (MoE) reward models seek to improve interpretability by routing prompts to specialized experts and characterizing experts through examples with high routing weights. However, routing weights only reveal which prompts an expert textitreceives, not how it textitjudges responses, providing only a partial account of expert behavior. We therefore propose Contribution-Contrast (CoCo) response-level interpretation, which faithfully characterizes experts' roles using chosen-rejected response pairs with the largest contribution contrasts, jointly capturing routing and preference behavior. Across automatic and human evaluations, CoCo yields more coherent, faithful, and specialized interpretations than router-based, score-based, and sparse autoencoder-based alternatives while maintaining competitive reward modeling accuracy. To the best of our knowledge, this is the first systematic study of interpretation methods for MoE reward models.