mlsanityData to 7 Oct 2026

Inference-Time Language Model Alignment via Integrated Value Guidance

arXiv 2409.17819 ↗First posted 26 Sep 2024cs.CL cs.AIcited by 19

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models are typically fine-tuned to align with human preferences, but tuning large models is computationally intensive and complex. In this work, we introduce textitIntegrated Value Guidance (IVG), a method that uses implicit and explicit value functions to guide language model decoding at token and chunk-level respectively, efficiently aligning large language models purely at inference time. This approach circumvents the complexities of direct fine-tuning and outperforms traditional methods. Empirically, we demonstrate the versatility of IVG across various tasks. In controlled sentiment generation and summarization tasks, our method significantly improves the alignment of large models using inference-time guidance from textttgpt2-based value functions. Moreover, in a more challenging instruction-following benchmark AlpacaEval 2.0, we show that both specifically tuned and off-the-shelf value functions greatly improve the length-controlled win rates of large models against textttgpt-4-turbo (e.g., 19.51% rightarrow 26.51% for textttMistral-7B-Instruct-v0.2 and 25.58% rightarrow 33.75% for textttMixtral-8x7B-Instruct-v0.1 with Tulu guidance).

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 36 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

24 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  2. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
9 more
  1. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  2. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  4. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  5. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  6. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  7. 2024Tuning Language Models by Proxycited by 88
  8. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  9. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!cited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 5.

19 papers.

Sort:

By area:

  1. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  2. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  3. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 33
  4. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  5. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 15
  6. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  7. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
  8. 2025Evolutionary Guided Decoding: Iterative Value Refinement for LLMscited by 3
  9. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 2
  10. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  11. 2026Learning Adaptive LLM Decodingcited by 2
  12. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1
7 more
  1. 2025Alignment-Aware Decodingcited by 1
  2. 2026To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blendingcited by 1
  3. 2025Mitigating Object Hallucination via Robust Local Perception Searchcited by 0
  4. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  5. 2026REAR: Test-time Preference Realignment through Reward Decompositioncited by 0
  6. 2026Robust Critics: Defending LLMs Against Multi-Turn Attackscited by 0
  7. 2026Beyond Token-Level Guidance: Inference-Time Alignment of Specialized LLMs via Cross-Family Representation Steeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
  2. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  3. 2024DeAL: Decoding-time Alignment for Large Language Models
  4. 2024Nudging: Inference-time Alignment of LLMs via Guided Decoding
  5. 2024InfAlign: Inference-aware language model alignment
  6. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
  7. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models
  8. 2024Aligner: Efficient Alignment by Learning to Correct