mlsanityData to 14 Oct 2026

The Asymptotics of Language Model Alignment with Memory

arXiv 2610.01828 ↗First posted 1 Oct 2026cs.CL cs.IT math.ITcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Language model (LM) alignment broadly aims to perturb a given LM Q into an aligned LM q such that i) the outputs produced by q and Q are 'close' in probability, ii) q has a higher expected reward than Q. Two common techniques for LM alignment are: KL-constrained RL, which requires knowledge of the LM distribution and is computationally expensive, and the best-of-n algorithm, which requires only sampling from the LM. The work of Yang et al. established asymptotic closeness between the distributions produced by the two alignment methods for an m--length i.i.d. token sequence output by the LM, in the limit as m increases to infinity. However, the i.i.d. assumption is not representative of practical LMs, whose output sequences often have memory. In this paper, we extend the asymptotic closeness result to the case when the m--length token sequence outputted by the LM is Markovian. Further, for finite-length output sequences -- particularly, when m=1 -- we provide a complete characterization of LM distributions and reward functions for which the KL-divergence between the distributions produced by the two alignment methods is zero -- a question first posed in Yang et al.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 28 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

15 of 28 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Variational Best-of-N Alignment8 Jul
  2. 2024Asymptotics of Language Model Alignment2 Apr
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 12

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  7. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  8. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  9. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 149
  10. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  11. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  12. 2024Large Language Models as Markov Chainscited by 35

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment of large language models with constrained learning
  2. 2024Efficient Alignment of Large Language Models via Data Sampling
  3. 2026Theoretical Limits of Language Model Alignment
  4. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  5. 2025Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation
  6. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling
  7. 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling
  8. 2026Why Self-Rewarding Works: Theoretical Guarantees for Iterative Alignment of Language Models