mlsanityData to 7 Oct 2026

The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization

arXiv 2609.39055 ↗First posted 30 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

How can we personalize a shared expert library from a user's pairwise choices? Prior work can realize different reward trade-offs by merging reward-specialized experts, given a vector of trade-off weights. In practice, users can more naturally choose between outputs than specify numerical weights. The challenge is therefore to turn these choices into the coefficients required for merging, while accounting for ambiguity when feedback is limited. Our key idea is to treat the unknown reward weights as latent variables: infer a posterior over them from pairwise choices and reward-score differences, and use its mean directly as the merge coefficients. We instantiate this idea as Bayesian Pairwise Merging (BPM), whose posterior also characterizes which reward trade-offs remain plausible given the feedback. We evaluate BPM on radiology summarization, image captioning, and story generation, spanning text-to-text and image-to-text generation. With 100 feedback per simulated persona, BPM achieves macro decided win rates of 91.7%, 77.1%, and 64.3% against uniform merge. For six pairs of simulated personas, each prefers the model fitted to its own feedback, a pattern also observed in a human proof-of-concept. In simulations under BPM's model and prior, its nominal 90% intervals for temperature-scaled reward weights achieve task-averaged marginal coverage of 88.9% and 89.2% with only 10 and 25 comparisons, respectively. BPM thus enables personalization from pairwise feedback without per-user policy training, while characterizing the coefficient ambiguity left by limited feedback.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 67 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

36 of 67 references.

References · 36

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2018Hierarchical Neural Story Generationcited by 1,182
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  8. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  9. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  10. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  11. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  12. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
24 more
  1. 2024GREEN: Generative Radiology Report Evaluation and Error Notationcited by 124
  2. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  3. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  4. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  5. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  6. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  7. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  8. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  9. 2018Learning to Summarize Radiology Findingscited by 54
  10. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  11. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  12. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 39
  13. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  14. 2025Language Model Personalization via Reward Factorizationcited by 27
  15. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  16. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  17. 2026Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Taskscited by 23
  18. 2024MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximationcited by 20
  19. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  20. 2024Pareto Merging: Multi-Objective Optimization for Preference-Aware Model Mergingcited by 13
  21. 2025Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generationcited by 13
  22. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
  23. 2026Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalizationcited by 1
  24. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1

Built on by

Not cited yet.