Multi-Stakeholder LLM Alignment: Decomposing Estimation from Aggregation
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Multi-stakeholder tasks require one output to satisfy users with conflicting preferences. Holistic LLM judges conflate utility estimation and utility aggregation, yielding unstable implicit weights. We show empirically and theoretically that this aggregation-specific weighting noise can create large score shifts when stakeholder satisfaction is dispersed; in our experiments, these weight-induced shifts also increase with stakeholder count. We propose textscDecompR: counterfactual-calibrated weights are fixed from query structure before candidate scoring, while per-role utilities are estimated independently, removing candidate-dependent weight drift and reducing estimation noise.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 33 references that are in the map. First posted on arXiv 26 May 2026.
Builds on
22 of 33 references.
Closest ancestors · 3
Same direction, newest first
- 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
- 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 19
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023Large Language Models are not Fair Evaluatorscited by 1,085
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
- 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
- 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
- 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
- 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
- 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
- 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
7 more
- 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
- 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
- 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
- 2026Your Group-Relative Advantage Is Biasedcited by 34
- 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 23
- 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 13
- 2026GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planningcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedback
- 2025Pluralistic Off-policy Evaluation and Alignment
- 2026Quantifying and Mitigating Self-Preference Bias of LLM Judges
- 2026Fairness Aware Reward Optimization
- 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluation
- 2025Value Drifts: Tracing Value Alignment During LLM Post-Training
- 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
- 2026Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation