mlsanityData to 14 Oct 2026

Unifying ICL, SFT, KL-Regularized RL Through a Bayesian Lens

arXiv 2609.05111 ↗First posted 4 Sep 2026cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Supervised fine-tuning (SFT), few-shot in-context learning (ICL), KL-regularized RLHF/RLVR, and on-policy distillation are usually treated as distinct post-training paradigms. We develop a unified Bayesian perspective in which each is an instance of a two-step template: construct a (generalized) Bayes or Gibbs posterior from a reference model and a utility signal (log-likelihood, reward, or advantage), then approximate it by a forward-KL projection onto a parametric family, either in-weights (SFT/RL) or in-context (ICL). This yields a single chain of equivalences: few-shot ICL is an amortized projection onto the Bayes posterior predictive, and reward-weighted SFT, reward-weighted ICL, and advantage-weighted SFT are forward-KL projections of reward-induced Gibbs posteriors. The framework explains why supervised warm-up is practically unavoidable for importance-weighted projections, and interprets R1/o1-style reasoning models as combining test-time Bayesian search with training-time amortization. Matched-budget experiments on Qwen3 models corroborate the picture: operators that share their learning-signal granularity produce nearly identical updates when support is good and diverge when it degrades, and reward-weighted projection performs on par with standard baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 27 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

20 of 27 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Reinforcement Learning from Human Feedback16 Apr
  2. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  8. 2015Policy Distillationcited by 440
  9. 2018Maximum a Posteriori Policy Optimisationcited by 328
  10. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  11. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  12. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
5 more
  1. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 110
  2. 2025Can Transformers Learn Full Bayesian Inference in Context?cited by 31
  3. 2025VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillationcited by 31
  4. 2025In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learningcited by 22
  5. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GIFT: Reconciling Post-Training Objectives via Variational Finite-Temperature Gibbs Initialization
  2. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoning
  3. 2025Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
  4. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
  5. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
  6. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
  7. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
  8. 2025Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models