mlsanityData to 14 Oct 2026

Exploring Domain Robust Lightweight Reward Models based on Router Mechanism

arXiv 2407.17546 ↗First posted 24 Jul 2024cs.LG cs.AI cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Recent advancements in large language models have heavily relied on the large reward model from reinforcement learning from human feedback for fine-tuning. However, the use of a single reward model across various domains may not always be optimal, often requiring retraining from scratch when new domain data is introduced. To address these challenges, we explore the utilization of small language models operating in a domain-specific manner based on router mechanisms. Our three approaches are: 1) utilize mixture of experts to form a single reward model by modularizing an internal router and experts, 2) employing external router to select the appropriate reward model from multiple domain-specific models, and 3) the framework reduces parameter size by loading reward models and router adapters onto a single small language model using adapters. Experimental validation underscores the effectiveness of our approach, demonstrating performance comparable to baseline methods while also reducing the total parameter size.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 29 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

24 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 982
11 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  2. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  3. 2023Large Language Model Routing with Benchmark Datasetscited by 172
  4. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  5. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  6. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  7. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  8. 2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
  9. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  10. 2024QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuningcited by 26
  11. 2023Towards Efficient and Effective Adaptation of Large Language Models for Sequential Recommendationcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N Samplingcited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2025Tiny Reward Models
  2. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  3. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
  4. 2025A Systematic Analysis of Base Model Choice for Reward Modeling
  5. 2025Generalist Reward Models: Found Inside Large Language Models
  6. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling
  8. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design