mlsanityData to 14 Oct 2026

Approximating Human Preferences Using a Multi-Judge Learned System

arXiv 2510.25884 ↗First posted 29 Oct 2025cs.AI cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Aligning LLM-based judges with human preferences is a significant challenge, as they are difficult to calibrate and often suffer from rubric sensitivity, bias, and instability. Overcoming this challenge advances key applications, such as creating reliable reward models for Reinforcement Learning from Human Feedback (RLHF) and building effective routing systems that select the best-suited model for a given user query. In this work, we propose a framework for modeling diverse, persona-based preferences by learning to aggregate outputs from multiple rubric-conditioned judges. We investigate the performance of this approach against naive baselines and assess its robustness through case studies on both human and LLM-judges biases. Our primary contributions include a persona-based method for synthesizing preference labels at scale and two distinct implementations of our aggregator: Generalized Additive Model (GAM) and a Multi-Layer Perceptron (MLP).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 47 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

26 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  4. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
9 more
  1. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  2. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  3. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  4. 2020How Interpretable and Trustworthy are GAMs?cited by 44
  5. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  6. 2024Serial Position Effects of Large Language Modelscited by 30
  7. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 30
  8. 2024A Little Human Data Goes A Long Waycited by 6
  9. 2025Beyond Monoliths: Expert Orchestration for More Capable, Democratic, and Safe Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Capturing Individual Human Preferences with Reward Features
  2. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling
  3. 2024PersonalLLM: Tailoring LLMs to Individual Preferences
  4. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modeling
  5. 2025Learning a Canonical Basis of Human Preferences from Binary Ratings
  6. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts