mlsanityData to 7 Oct 2026

BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback

arXiv 2402.02479 ↗First posted 4 Feb 2024cs.LG cs.AI cs.CLcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Distribution matching methods for language model alignment such as Generation with Distributional Control (GDC) and Distributional Policy Gradient (DPG) have not received the same level of attention in reinforcement learning from human feedback (RLHF) as contrastive methods such as Sequence Likelihood Calibration (SLiC), Direct Preference Optimization (DPO) and its variants. We identify high variance of the gradient estimate as the primary reason for the lack of success of these methods and propose a self-normalized baseline to reduce the variance. We further generalize the target distribution in DPG, GDC and DPO by using Bayes' rule to define the reward-conditioned posterior. The resulting approach, referred to as BRAIn - Bayesian Reward-conditioned Amortized Inference acts as a bridge between distribution matching methods and DPO and significantly outperforms prior art in summarization and Antropic HH tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 37 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

25 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2024Mixtral of Expertscited by 1,785
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  10. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  12. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
8 more
  1. 2023Pretraining Language Models with Human Preferencescited by 245
  2. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  3. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  4. 2022Defining and Characterizing Reward Hackingcited by 184
  5. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  6. 2020A Distributional Approach to Controlled Text Generationcited by 117
  7. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  8. 2019Distributional Reinforcement Learning for Energy-Based Sequential Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  2. 2024Nemotron-4 340B Technical Reportcited by 143
  3. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignmentcited by 10
  4. 2025Optimal Policy Minimum Bayesian Riskcited by 2
  5. 2025PIPA: Preference Alignment as Prior-Informed Statistical Estimationcited by 1
  6. 2026DKL: Decoupled Knowledge Learning for Instruction-Tuned Language Modelscited by 1
  7. 2026DIVERGE: Diversity-Enhanced RAG for Open-Ended Information Seekingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bayesian Reward Models for LLM Alignment
  2. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  3. 2023Aligning Language Models with Human Preferences via a Bayesian Approach
  4. 2023Amortizing intractable inference in large language models
  5. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  6. 2024Aligning Crowd Feedback via Distributional Preference Reward Modeling
  7. 2024Aligning language models with human preferences
  8. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms