mlsanityData to 7 Oct 2026

DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging

arXiv 2407.01470 ↗First posted 1 Jul 2024cs.CLcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a popular strategy for aligning large language models (LLMs) with desired behaviors. Reward modeling is a crucial step in RLHF. However, collecting paired preference data for training reward models is often costly and time-consuming, especially for domain-specific preferences requiring expert annotation. To address this challenge, we propose the Domain knowledge merged Reward Model (DogeRM), a novel framework that integrates domain-specific knowledge into a general reward model by model merging. The experiments demonstrate that DogeRM enhances performance across different benchmarks and provide a detailed analysis showcasing the effects of model merging, showing the great potential of facilitating model alignment.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 71 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

51 of 71 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  2. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards7 Jun
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  7. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 43

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
31 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  13. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  14. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  15. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  16. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  17. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  18. 2022Teaching language models to support answers with verified quotescited by 290
  19. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  20. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  21. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  22. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  23. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  24. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  25. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  26. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  27. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  28. 2024MAmmoTH2: Scaling Instructions from the Webcited by 143
  29. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  30. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languagescited by 52
  31. 2023Won't Get Fooled Again: Answering Questions with False Premisescited by 47

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  3. 2025Transferring Textual Preferences to Vision-Language Understanding through Model Mergingcited by 3
  4. 2025Brittleness and Promise: Knowledge Graph Based Reward Modeling for Diagnostic Reasoningcited by 3
  5. 2025Optimal Brain Iterative Merging: Mitigating Interference in LLM Mergingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  3. 2026Real-Time Aligned Reward Model beyond Semantics
  4. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  5. 2025Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
  6. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  7. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
  8. 2024Optimal Design for Reward Modeling in RLHF