mlsanityData to 14 Oct 2026

A Probability--Quality Trade-off in Aligned Language Models and its Relation to Sampling Adaptors

arXiv 2406.10203 ↗First posted 14 Jun 2024cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The relationship between the quality of a string, as judged by a human reader, and its probability, p(boldsymboly) under a language model undergirds the development of better language models. For example, many popular algorithms for sampling from a language model have been conceived with the goal of manipulating p(boldsymboly) to place higher probability on strings that humans deem of high quality. In this article, we examine the probability--quality relationship in language models explicitly aligned to human preferences, e.g., through reinforcement learning through human feedback. We show that, when sampling corpora from an aligned language model, there exists a trade-off between the strings' average reward and average log-likelihood under the prior language model, i.e., the same model before alignment with human preferences. We provide a formal treatment of this phenomenon and demonstrate how a choice of sampling adaptor allows for a selection of how much likelihood we exchange for the reward.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 60 references that are in the map. First posted on arXiv 14 Jun 2024.

Builds on

36 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  2. 2024Direct Preference Optimization with an Offset16 Feb
  3. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model23 Jan
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2018Hierarchical Neural Story Generationcited by 1,193
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  11. 2022Red Teaming Language Models with Language Modelscited by 1,026
  12. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
19 more
  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  2. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  3. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  4. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  5. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  6. 2017Toward Controlled Generation of Textcited by 272
  7. 2022A Contrastive Framework for Neural Text Generationcited by 225
  8. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  9. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  10. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  11. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  12. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 96
  13. 2022On Decoding Strategies for Neural Text Generatorscited by 90
  14. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  15. 2020Consistency of a Recurrent Language Model With Respect to Incomplete Decodingcited by 54
  16. 2022A Measure-Theoretic Characterization of Tight Language Modelscited by 33
  17. 2023On the Efficacy of Sampling Adapterscited by 17
  18. 2024Simpson's Paradox and the Accuracy-Fluency Tradeoff in Translationcited by 8
  19. 2017Convergence diagnostics for MCMC draws of a categorical variablecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning language models with human preferences
  2. 2026Theoretical Limits of Language Model Alignment
  3. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  4. 2025Soft Best-of-n Sampling for Model Alignment
  5. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  6. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment
  7. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignment
  8. 2024Transforming and Combining Rewards for Aligning Large Language Models