mlsanityData to 7 Oct 2026

DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling

arXiv 2403.01197 ↗First posted 2 Mar 2024cs.CLcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

The performance of the reward model (RM) is a critical factor in improving the effectiveness of the large language model (LLM) during alignment fine-tuning. There remain two challenges in RM training: 1) training the same RM using various categories of data may cause its generalization performance to suffer from multi-task disturbance, and 2) the human annotation consistency rate is generally only 60% to 75%, causing training data to contain a lot of noise. To tackle these two challenges, we introduced the idea of Mixture-of-Experts (MoE) into the field of RM for the first time. We propose the Double-Layer MoE RM (DMoERM). The outer layer MoE is a sparse model. After classifying an input into task categories, we route it to the corresponding inner layer task-specific model. The inner layer MoE is a dense model. We decompose the specific task into multiple capability dimensions and individually fine-tune a LoRA expert on each one. Their outputs are then synthesized by an MLP to compute the final rewards. To minimize costs, we call a public LLM API to obtain the capability preference labels. The validation on manually labeled datasets confirms that our model attains superior consistency with human preference and outstrips advanced generative approaches. Meanwhile, through BoN sampling and RL experiments, we demonstrate that our model outperforms state-of-the-art ensemble methods of RM and mitigates the overoptimization problem. Our code and dataset are available at: https://github.com/quanshr/DMoERM-v1.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 48 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

33 of 48 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  2. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  5. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  6. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  7. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  9. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  11. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  11. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  12. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
9 more
  1. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  2. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  3. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  4. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  5. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  6. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  7. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  8. 2021Go Wider Instead of Deepercited by 56
  9. 2023ChatHome: Development and Evaluation of a Domain-Specific Language Model for Home Renovationcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 5; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024A Survey on LoRA of Large Language Modelscited by 67
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  4. 2024Language Models can Self-Lengthen to Generate Long Textscited by 24
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  6. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  7. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  8. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  9. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 9
  10. 2024LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Banditscited by 7
  11. 2024Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularitycited by 4
  12. 2026Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modelingcited by 2
3 more
  1. 2026Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RLcited by 1
  2. 2025UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hackingcited by 0
  3. 2026Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrastcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models
  3. 2026Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
  4. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  5. 2025Advancing Expert Specialization for Better MoE
  6. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  7. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  8. 2024A Survey on Mixture of Experts in Large Language Models