mlsanityData to 7 Oct 2026

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

arXiv 2406.12845 ↗First posted 18 Jun 2024cs.LG cs.CLcited by 369

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has emerged as the primary method for aligning large language models (LLMs) with human preferences. The RLHF process typically starts by training a reward model (RM) using human preference data. Conventional RMs are trained on pairwise responses to the same user request, with relative ratings indicating which response humans prefer. The trained RM serves as a proxy for human preferences. However, due to the black-box nature of RMs, their outputs lack interpretability, as humans cannot intuitively understand why an RM thinks a response is good or not. As RMs act as human preference proxies, we believe they should be human-interpretable to ensure that their internal decision processes are consistent with human preferences and to prevent reward hacking in LLM alignment. To build RMs with interpretable preferences, we propose a two-stage approach: i) train an Absolute-Rating Multi-Objective Reward Model (ArmoRM) with multi-dimensional absolute-rating data, each dimension corresponding to a human-interpretable objective (e.g., honesty, verbosity, safety); ii) employ a Mixture-of-Experts (MoE) strategy with a gating network that automatically selects the most suitable reward objectives based on the context. We efficiently trained an ArmoRM with Llama-3 8B and a gating network consisting of a shallow MLP on top of the ArmoRM. Our trained model, ArmoRM-Llama3-8B, obtains state-of-the-art performance on RewardBench, a benchmark evaluating RMs for language modeling. Notably, the performance of our model surpasses the LLM-as-a-judge method with GPT-4 judges by a margin, and approaches the performance of the much larger Nemotron-4 340B reward model.

Key facts. Cited by 369 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 60 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

49 of 60 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF29 Apr
  4. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  5. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  6. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  7. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  9. 2023Nash Learning from Human Feedback1 Dec
  10. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models16 Oct
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  12. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
7 more
  1. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 30

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
18 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  2. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  3. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  4. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  5. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  6. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  7. 2022Defining and Characterizing Reward Hackingcited by 184
  8. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  9. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  10. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  11. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  12. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
  13. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  14. 2021Leveraging Sparse Linear Layers for Debuggable Deep Networkscited by 82
  15. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  16. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  17. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  18. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 74; 2025: 192; 2026: 103.

369 papers.

Sort:

By area:

  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  4. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  5. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  6. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  7. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  8. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  9. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  10. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  11. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 113
  12. 2024Critique-out-Loud Reward Modelscited by 105
38 more
  1. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  3. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 89
  4. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 89
  5. 2024How to Evaluate Reward Models for RLHFcited by 84
  6. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  7. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  8. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  9. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
  10. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  11. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  12. 2024Self-Taught Evaluatorscited by 70
  13. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  14. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  15. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  16. 2024Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generationcited by 64
  17. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  18. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  19. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  20. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  21. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
  22. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  23. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  24. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 57
  25. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  26. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  27. 2024Self-Consistency Preference Optimizationcited by 50
  28. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  29. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  30. 2025Diverse Preference Optimizationcited by 48
  31. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  32. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  33. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  34. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
  35. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  36. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  37. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  38. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34

50 most cited of 369 All 369 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Influencing Humans to Conform to Preference Models for RLHF
  2. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  3. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  5. 2024Towards Comprehensive Preference Data Collection for Reward Modeling
  6. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2026In-Context Reward Adaptation for Robust Preference Modeling