mlsanityData to 7 Oct 2026

Transfer Q Star: Principled Decoding for LLM Alignment

arXiv 2405.20495 ↗First posted 30 May 2024cs.CL cs.LGcited by 46

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning foundation models is essential for their safe and trustworthy deployment. However, traditional fine-tuning methods are computationally intensive and require updating billions of model parameters. A promising alternative, alignment via decoding, adjusts the response distribution directly without model updates to maximize a target reward r, thus providing a lightweight and adaptable framework for alignment. However, principled decoding methods rely on oracle access to an optimal Q-function (Q^*), which is often unavailable in practice. Hence, prior SoTA methods either approximate this Q^* using Q^π_textttsft (derived from the reference textttSFT model) or rely on short-term rewards, resulting in sub-optimal decoding performance. In this work, we propose Transfer Q^*, which implicitly estimates the optimal value function for a target reward r through a baseline model ρ_textttBL aligned with a baseline reward ρ_textttBL (which can be different from the target reward r). Theoretical analyses of Transfer Q^* provide a rigorous characterization of its optimality, deriving an upper bound on the sub-optimality gap and identifying a hyperparameter to control the deviation from the pre-trained reference textttSFT model based on user needs. Our approach significantly reduces the sub-optimality gap observed in prior SoTA methods and demonstrates superior empirical performance across key metrics such as coherence, diversity, and quality in extensive tests on several synthetic and real datasets.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 45 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

33 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  2. 2024Decoding-time Realignment of Language Models5 Feb
  3. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 28

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Mistral 7Bcited by 3,065
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2018Hierarchical Neural Story Generationcited by 1,182
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
16 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  5. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  7. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  8. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  9. 2022A Contrastive Framework for Neural Text Generationcited by 224
  10. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  11. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  12. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  13. 2023Controlled Decoding from Language Modelscited by 133
  14. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  15. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  16. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 26; 2026: 10.

46 papers.

Sort:

By area:

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  3. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  4. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  5. 2024MegaAgent: A Large-Scale Autonomous LLM-based Multi-Agent System Without Predefined SOPscited by 45
  6. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  7. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  8. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  9. 2024InfAlign: Inference-aware language model alignmentcited by 34
  10. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  11. 2024Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehensioncited by 23
  12. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
34 more
  1. 2025Collab: Controlled Decoding using Mixture of Agents for LLM Alignmentcited by 13
  2. 2025ScalingNoise: Scaling Inference-Time Search for Generating Infinite Videoscited by 12
  3. 2025Speeding up Speculative Decoding via Sequential Approximate Verificationcited by 11
  4. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  5. 2025TRACE Back from the Future: A Probabilistic Reasoning Approach to Controllable Language Generationcited by 8
  6. 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Timecited by 8
  7. 2025Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Expertscited by 8
  8. 2024Guaranteed Generation from Large Language Modelscited by 7
  9. 2025MIRA: Towards Mitigating Reward Hacking in Inference-Time Alignment of T2I Diffusion Modelscited by 7
  10. 2025Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examplescited by 6
  11. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5
  12. 2024On the Low-Rank Parametrization of Reward Models for Controlled Language Generationcited by 3
  13. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  14. 2025Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Modelscited by 2
  15. 2025Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Modelscited by 2
  16. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  17. 2025Tokenized Bandit for LLM Decoding and Alignmentcited by 2
  18. 2025Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioningcited by 2
  19. 2025Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modelingcited by 2
  20. 2025Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editingcited by 2
  21. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  22. 2026Decoding as Optimisation on the Probability Simplex: From Top-K to Top-P (Nucleus) to Best-of-K Samplerscited by 2
  23. 2026Inference-time Alignment via Sparse Junction Steeringcited by 2
  24. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  25. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1
  26. 2025PROPS: Progressively Private Self-alignment of Large Language Modelscited by 1
  27. 2026TARo: Token-level Adaptive Routing for LLM Test-time Alignmentcited by 1
  28. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  29. 2025An Empirical Study of Sample Selection Strategies for Large Language Model Repaircited by 0
  30. 2025Test-time reward-guided alignment of language models by importance sampling on pre-logit spacecited by 0
  31. 2026CLIPer: Tailoring Diverse User Preference via Classifier-Guided Inference-Time Personalizationcited by 0
  32. 2026C3-Bench: A Context-Aware Change Captioning Benchmarkcited by 0
  33. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
  34. 2026Composable Decoding on the Probability Simplex: Theory and Implementationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Nudging: Inference-time Alignment of LLMs via Guided Decoding
  2. 2024Fast Best-of-N Decoding via Speculative Rejection
  3. 2024Decoding-Time Language Model Alignment with Multiple Objectives
  4. 2026AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation
  5. 2026Safe Inference-Time Alignment via Lagrangian Reward Augmentation
  6. 2024Aligning Large Language Models with Representation Editing: A Control Perspective
  7. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inference
  8. 2025Alignment-Aware Decoding