mlsanityData to 7 Oct 2026

Bayesian Preference Learning for Test-Time Steerable Reward Models

arXiv 2602.08819 ↗First posted 9 Feb 2026cs.LG cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are central to aligning language models with human preferences via reinforcement learning (RL). As RL is increasingly applied to settings such as verifiable rewards and multi-objective alignment, RMs are expected to encode more complex and multifaceted preference distributions. However, classifier RMs remain static once trained, limiting their adaptability at test time. We propose Variational In-Context Reward Modeling (ICRM), a novel Bayesian reward modeling objective that enables test-time steerability via in-context preference demonstrations. ICRM casts reward modeling as amortized variational inference over a latent preference probability under the Bradley-Terry model using a conjugate Beta prior. We show that ICRM adapts to unseen preference distributions at test time for both single and multi-objective settings. With more demonstrations, ICRM improves RM-Bench accuracy from 60.5 to 70.8, achieves lower calibration error than a generative judge on moral dilemma preferences, and expands the attainable Pareto frontier under conflicting preferences. We further study the practical applicability of ICRM for RL training, showing that it can effectively encode verifiable rewards by outperforming a conventional RM in math reasoning. Finally, we provide theoretical guarantees that the variational objective admits a global interior optimum with finite confidence, and we analyze how KL regularization mitigates reward over-optimization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 66 of its 81 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

66 of 81 references.

Closest ancestors · 24

Same direction, newest first

  1. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoning2 Sep
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  3. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages16 May
  4. 2025On the Robustness of Reward Models for Language Model Alignment12 May
  5. 2025Energy-Based Reward Models for Robust Language Model Alignment17 Apr
  6. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  7. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  8. 2024Generative Reward Models2 Oct
  9. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  10. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  11. 2024Critique-out-Loud Reward Models21 Aug
  12. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
12 more
  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  2. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  3. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization24 Mar
  4. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  5. 2024Bayesian Reward Models for LLM Alignment20 Feb
  6. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  7. 2024Transforming and Combining Rewards for Aligning Large Language Models1 Feb
  8. 2024Self-Rewarding Language Models18 Jan
  9. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 42

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
30 more
  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  4. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  5. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  7. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  8. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  10. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  11. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  12. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  13. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 123
  14. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  15. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  16. 2024Active Preference Learning for Large Language Modelscited by 55
  17. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  18. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  19. 2024ICLR: In-Context Learning of Representationscited by 49
  20. 2024LLM Voting: Human Choices and AI Collective Decision Makingcited by 47
  21. 2025Learning without training: The implicit dynamics of in-context learningcited by 41
  22. 2019The continuous Bernoulli: fixing a pervasive error in variational autoencoderscited by 40
  23. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 39
  24. 2020Being Bayesian about Categorical Probabilitycited by 35
  25. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  26. 2024Reward Centeringcited by 16
  27. 2024Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedbackcited by 15
  28. 2025AlphaPO: Reward Shape Matters for LLM Alignmentcited by 14
  29. 2022A Bayesian Bradley-Terry model to compare multiple ML algorithms on multiple data setscited by 9
  30. 2023Generalized Bradley-Terry Models for Score Estimation from Paired Comparisonscited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport
  2. 2026In-Context Reward Adaptation for Robust Preference Modeling
  3. 2026Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling
  4. 2025PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
  5. 2026Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling
  6. 2025Preference Learning for AI Alignment: a Causal Perspective
  7. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective