mlsanityData to 14 Oct 2026

UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking

arXiv 2512.00724 ↗First posted 30 Nov 2025cs.LG cs.IRcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reward models (RMs) are critical to reinforcement learning from human feedback (RLHF). However, policy models can exploit biases or spurious correlations in dense RMs, causing reward hacking: RM scores rise while alignment with human preferences deteriorates, especially under distribution shift. We propose UMM-RM (Upcycle-and-Merge MoE Reward Model), which copies the feed-forward layers of an SFT checkpoint to initialize shared and routed experts, then trains the model on preference data. The always-active shared expert captures instruction-agnostic preference signals, while routed experts model fine-grained preferences across instructions or tasks. The experts are then merged into a single dense RM using learnable weights, preserving the robustness and exploitation resistance of expert diversity without MoE or ensemble inference overhead. Across multiple base models and preference datasets, UMM-RM improves preference accuracy, reduces reward hacking during PPO training, and achieves more stable preference alignment than standard dense RMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 32 references that are in the map. First posted on arXiv 30 Nov 2025.

Builds on

31 of 32 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation14 Nov
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  3. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  4. 2024Scalable Ensembling For Mitigating Reward Overoptimisation3 Jun
  5. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance7 Apr
  6. 2024Fine-Tuning Language Models with Reward Learning on Policy28 Mar
  7. 2024DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling2 Mar
  8. 2024Generalizing Reward Modeling for Out-of-Distribution Preference Learning22 Feb
  9. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling14 Feb
  10. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  11. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  12. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
4 more
  1. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  11. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  12. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
3 more
  1. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  2. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  3. 2024XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Expertscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RMB: Reward Model Boosting Mitigates Reward Hacking
  2. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
  3. 2024RRM: Robust Reward Model Training Mitigates Reward Hacking
  4. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking
  5. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling
  6. 2025Rectifying Shortcut Behaviors in Preference-based Reward Learning
  7. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  8. 2026Uncertainty-Aware Reward Modeling for Stable RLHF