mlsanityData to 13 Oct 2026

ARGS: Alignment as Reward-Guided Search

arXiv 2402.01694 ↗First posted 23 Jan 2024cs.CL cs.AI cs.LGcited by 102

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models with human objectives is paramount, yet common approaches including RLHF suffer from unstable and resource-intensive training. In response to this challenge, we introduce ARGS, Alignment as Reward-Guided Search, a novel framework that integrates alignment into the decoding process, eliminating the need for expensive RL training. By adjusting the model's probabilistic predictions using a reward signal, ARGS generates texts with semantic diversity while being aligned with human preferences, offering a promising and flexible solution for aligning language models. Notably, ARGS demonstrates consistent enhancements in average reward compared to baselines across diverse alignment tasks and various model dimensions. For example, under the same greedy-based decoding strategy, our method improves the average reward by 19.56% relative to the baseline and secures a preference or tie score of 64.33% in GPT-4 evaluation. We believe that our framework, emphasizing decoding-time alignment, paves the way for more responsive language models in the future. Code is publicly available at: urlhttps://github.com/deeplearning-wisc/args.

Key facts. Cited by 102 papers in the map (data through 13 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 102 of its 160 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

102 of 160 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Preference Ranking Optimization for Human Alignment30 Jun
  2. 2022Training language models to follow instructions with human feedback4 Mar
  3. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 99

  1. 2017Attention Is All You Needcited by 60,106
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  7. 2017Deep reinforcement learning from human preferencescited by 3,871
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  9. 2023Let's Verify Step by Stepcited by 3,672
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,181
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,893
38 more
  1. 2020Learning to summarize from human feedbackcited by 2,575
  2. 2022Emergent Abilities of Large Language Modelscited by 2,499
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,736
  4. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,327
  5. 2020Energy-based Out-of-distribution Detectioncited by 1,297
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,249
  7. 2018Hierarchical Neural Story Generationcited by 1,186
  8. 2021Ethical and social risks of harm from Language Modelscited by 964
  9. 2021Generalized Out-of-Distribution Detection: A Surveycited by 799
  10. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 797
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 751
  12. 2017Deep Reinforcement Learning that Matterscited by 734
  13. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 680
  14. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  15. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 600
  16. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 598
  17. 2016Self-critical Sequence Training for Image Captioningcited by 544
  18. 2022Improving alignment of dialogue agents via targeted human judgementscited by 529
  19. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  20. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 459
  21. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 437
  22. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  23. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  24. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 375
  25. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  26. 2017Snapshot Ensembles: Train 1, get M for freecited by 350
  27. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 339
  28. 2022VOS: Learning What You Don't Know by Virtual Outlier Synthesiscited by 300
  29. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  30. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  31. 2023Challenges and Applications of Large Language Modelscited by 253
  32. 2022The Alignment Problem from a Deep Learning Perspectivecited by 252
  33. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 251
  34. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  35. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  36. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 228
  37. 2022A Contrastive Framework for Neural Text Generationcited by 224
  38. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 19; 2025: 56; 2026: 27.

102 papers.

Sort:

By area:

  1. 2025TTRL: Test-Time Reinforcement Learningcited by 262
  2. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  3. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 93
  4. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 93
  5. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 85
  6. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 64
  7. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  8. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 59
  9. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  10. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  11. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  12. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 44
38 more
  1. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  2. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 41
  3. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 39
  4. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 39
  5. 2025Video-T1: Test-Time Scaling for Video Generationcited by 39
  6. 2025A Survey of Multilingual Reasoning in Language Modelscited by 32
  7. 2025MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPOcited by 30
  8. 2025Language Model Personalization via Reward Factorizationcited by 29
  9. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 27
  10. 2025Virtual Agent Economiescited by 18
  11. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  12. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 14
  13. 2025Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generationcited by 14
  14. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Modelscited by 13
  15. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferencescited by 13
  16. 2025IndustryEQA: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarioscited by 12
  17. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 12
  18. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 12
  19. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  20. 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularizationcited by 11
  21. 2024Probing the Safety Response Boundary of Large Language Models via Unsafe Decoding Path Generationcited by 10
  22. 2024RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistributioncited by 10
  23. 2025Temporal Consistency for LLM Reasoning Process Error Identificationcited by 10
  24. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  25. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
  26. 2025AdaBoN: Adaptive Best-of-N Alignmentcited by 9
  27. 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Timecited by 9
  28. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
  29. 2026Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Controlcited by 9
  30. 2025Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Expertscited by 8
  31. 2024DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignmentcited by 7
  32. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decodingcited by 7
  33. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  34. 2025BiasFilter: An Inference-Time Debiasing Framework for Large Language Modelscited by 6
  35. 2025Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignmentcited by 6
  36. 2025Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examplescited by 6
  37. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 6
  38. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Timecited by 5

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  2. 2025Energy-Based Reward Models for Robust Language Model Alignment
  3. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  4. 2026Gradient-Guided Reward Optimization for Inference-time Alignment
  5. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms
  6. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
  7. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment
  8. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment