# ARGS: Alignment as Reward-Guided Search

arXiv: https://arxiv.org/abs/2402.01694 · HTML: https://mlsanity.com/paper/2402.01694/

## Key facts

- Cited by 103 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 102 of its 160 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 102)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)

## Built on by (most cited first, 30 of 103)

- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Fast Best-of-N Decoding via Speculative Rejection](https://mlsanity.com/paper/2410.20290/) (2024)
- [Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment](https://mlsanity.com/paper/2503.21878/) (2025)
- [Test-time Alignment of Diffusion Models without Reward Over-optimization](https://mlsanity.com/paper/2501.05803/) (2025)
- [Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review](https://mlsanity.com/paper/2501.09685/) (2025)
- [Aligning Large Language Models with Representation Editing: A Control Perspective](https://mlsanity.com/paper/2406.05954/) (2024)
- [TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling](https://mlsanity.com/paper/2410.16033/) (2024)
- [PAD: Personalized Alignment of LLMs at Decoding-Time](https://mlsanity.com/paper/2410.04070/) (2024)
- [ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time](https://mlsanity.com/paper/2410.06625/) (2024)
- [DeAL: Decoding-time Alignment for Large Language Models](https://mlsanity.com/paper/2402.06147/) (2024)
- [GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment](https://mlsanity.com/paper/2410.08193/) (2024)
- [Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment](https://mlsanity.com/paper/2411.18688/) (2024)
- [A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications](https://mlsanity.com/paper/2503.17003/) (2025)
- [Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback](https://mlsanity.com/paper/2501.12895/) (2025)
- [Video-T1: Test-Time Scaling for Video Generation](https://mlsanity.com/paper/2503.18942/) (2025)
- [Cascade Reward Sampling for Efficient Decoding-Time Alignment](https://mlsanity.com/paper/2406.16306/) (2024)
- [Evaluation of Best-of-N Sampling Strategies for Language Model Alignment](https://mlsanity.com/paper/2502.12668/) (2025)
- [A Survey of Multilingual Reasoning in Language Models](https://mlsanity.com/paper/2502.09457/) (2025)
- [Language Model Personalization via Reward Factorization](https://mlsanity.com/paper/2503.06358/) (2025)
- [MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO](https://mlsanity.com/paper/2505.13031/) (2025)
- [A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models](https://mlsanity.com/paper/2504.07070/) (2025)
- [Virtual Agent Economies](https://mlsanity.com/paper/2509.10147/) (2025)
- [Representation-Based Exploration for Language Models: From Test-Time to Post-Training](https://mlsanity.com/paper/2510.11686/) (2025)
- [On Almost Surely Safe Alignment of Large Language Models at Inference-Time](https://mlsanity.com/paper/2502.01208/) (2025)
- [Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generation](https://mlsanity.com/paper/2502.10762/) (2025)
- [Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models](https://mlsanity.com/paper/2411.08733/) (2024)
- [Drift: Decoding-time Personalized Alignments with Implicit User Preferences](https://mlsanity.com/paper/2502.14289/) (2025)
- [IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios](https://mlsanity.com/paper/2505.20640/) (2025)
- [Multi-objective Large Language Model Alignment with Hierarchical Experts](https://mlsanity.com/paper/2505.20925/) (2025)
- [On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation](https://mlsanity.com/paper/2603.22117/) (2026)

## Abstract (arXiv)

Aligning large language models with human objectives is paramount, yet common approaches including RLHF suffer from unstable and resource-intensive training. In response to this challenge, we introduce ARGS, Alignment as Reward-Guided Search, a novel framework that integrates alignment into the decoding process, eliminating the need for expensive RL training. By adjusting the model's probabilistic predictions using a reward signal, ARGS generates texts with semantic diversity while being aligned with human preferences, offering a promising and flexible solution for aligning language models. Notably, ARGS demonstrates consistent enhancements in average reward compared to baselines across diverse alignment tasks and various model dimensions. For example, under the same greedy-based decoding strategy, our method improves the average reward by 19.56% relative to the baseline and secures a preference or tie score of 64.33% in GPT-4 evaluation. We believe that our framework, emphasizing decoding-time alignment, paves the way for more responsive language models in the future. Code is publicly available at: urlhttps://github.com/deeplearning-wisc/args.