Data to 7 Oct 2026

Value Augmented Sampling for Language Model Alignment and Personalization

arXiv 2405.06639 ↗First posted 10 May 2024cs.LG cs.AI cs.CLcited by 49

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more

LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Aligning Large Language Models (LLMs) to cater to different human preferences, learning new skills, and unlearning harmful behavior is an important problem. Search-based methods, such as Best-of-N or Monte-Carlo Tree Search, are performant, but impractical for LLM adaptation due to their high inference cost. On the other hand, using Reinforcement Learning (RL) for adaptation is computationally efficient, but performs worse due to the optimization challenges in co-training the value function and the policy. We present a new framework for reward optimization, Value Augmented Sampling (VAS), that can maximize different reward functions using data sampled from only the initial, frozen LLM. VAS solves for the optimal reward-maximizing policy without co-training the policy and the value function, making the optimization stable, outperforming established baselines, such as PPO and DPO, on standard benchmarks, and achieving comparable results to Best-of-128 with lower inference cost. Unlike existing RL methods that require changing the weights of the LLM, VAS does not require access to the weights of the pre-trained LLM. Thus, it can even adapt LLMs (e.g., ChatGPT), which are available only as APIs. In addition, our algorithm unlocks the new capability of composing several rewards and controlling the extent of each one during deployment time, paving the road ahead for the future of aligned, personalized LLMs.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 52 references that are in the map. First posted on arXiv 10 May 2024.

Builds on

35 of 52 references.

Closest ancestors · 1

Same direction, newest first

2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 34

2017Proximal Policy Optimization Algorithmscited by 14,441
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769

22 more

2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
2021Recursively Summarizing Books with Human Feedbackcited by 287
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
2022The Primacy Bias in Deep Reinforcement Learningcited by 242
2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
2023Controlled Decoding from Language Modelscited by 133
2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
2023Don't lose the message while paraphrasing: A study on content preserving style transfercited by 25
2023VA-learning as a more efficient alternative to Q-learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 25; 2026: 10.

49 papers.

Sort:

By area:

2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 67
2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 36
2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34

37 more

2025Language Model Personalization via Reward Factorizationcited by 27
2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 17
2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 14
2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
2025General Intelligence Requires Reward-based Pretrainingcited by 12
2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 12
2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
2024Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generationcited by 10
2024Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspectivecited by 9
2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
2025Pretrain Value, Not Reward: Decoupled Value Policy Optimizationcited by 6
2026Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoningcited by 6
2025Towards Cost-Effective Reward Guided Text Generationcited by 5
2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5
2026Positive Alignment: Artificial Intelligence for Human Flourishingcited by 5
2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignmentcited by 4
2024On the Low-Rank Parametrization of Reward Models for Controlled Language Generationcited by 3
2025Evolutionary Guided Decoding: Iterative Value Refinement for LLMscited by 3
2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
2026EXACT: Explicit Attribute-Guided Decoding-Time Personalizationcited by 3
2025Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMscited by 2
2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 2
2025PITA: Preference-Guided Inference-Time Alignment for LLM Post-Trainingcited by 2
2025Controlling Multimodal LLMs via Reward-guided Decodingcited by 2
2025Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editingcited by 2
2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1
2025Generating Fair Consensus Statements with Social Choice on Token-Level MDPscited by 1
2026Safe Inference-Time Alignment via Lagrangian Reward Augmentationcited by 1
2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
2025Test-time reward-guided alignment of language models by importance sampling on pre-logit spacecited by 0
2026Selective Safety Steering via Value-Filtered Decodingcited by 0
2026Inverse RL Helps Align AI by Imitating Humanscited by 0
2026ExpertIVS: Sociological Expert Driven Individual Value Simulation in Large Language Modelscited by 0
2026When Reasoning Helps Action: Monitoring and Steering Chain-of-Thought in Vision-Language-Action Policiescited by 0

Similar, not cited

Close in wording, no citation either way