Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
also in 1 more
Abstract (arXiv)
Reward models are central to learning from human preferences, yet identifying what drives their predictions remains challenging. Recent sparse Mixture-of-Experts (MoE) reward models seek to improve interpretability by routing prompts to specialized experts and characterizing experts through examples with high routing weights. However, routing weights only reveal which prompts an expert textitreceives, not how it textitjudges responses, providing only a partial account of expert behavior. We therefore propose Contribution-Contrast (CoCo) response-level interpretation, which faithfully characterizes experts' roles using chosen-rejected response pairs with the largest contribution contrasts, jointly capturing routing and preference behavior. Across automatic and human evaluations, CoCo yields more coherent, faithful, and specialized interpretations than router-based, score-based, and sparse autoencoder-based alternatives while maintaining competitive reward modeling accuracy. To the best of our knowledge, this is the first systematic study of interpretation methods for MoE reward models.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 25 references that are in the map. First posted on arXiv 31 Jul 2026.
Builds on
19 of 25 references.
Closest ancestors · 10
Same direction, newest first
- 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling2 Jun
- 2025SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder11 Nov
- 2025Interpretable Reward Model via Sparse Autoencoder12 Aug
- 2025Reward Models in Deep Reinforcement Learning: A Survey18 Jun
- 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learning30 May
- 2025Rethinking Diverse Human Preference Learning through Principal Component Analysis18 Feb
- 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning19 Aug
- 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
- 2024DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling2 Mar
- 2024MaxMin-RLHF: Alignment with Diverse Human Preferences14 Feb
Other references · 9
- 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
- 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
- 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
- 2024LLMs + Persona-Plug = Personalized LLMscited by 32
- 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
- 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
- 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 19
- 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
- 2025CoPL: Collaborative Preference Learning for Personalizing LLMscited by 10
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers
- 2025Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
- 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
- 2026TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptation
- 2026Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Experts
- 2026RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing
- 2025Reward Model Interpretability via Optimal and Pessimal Tokens
- 2026The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models