mlsanityData to 7 Oct 2026

DeAL: Decoding-time Alignment for Large Language Models

arXiv 2402.06147 ↗First posted 5 Feb 2024cs.AI cs.CLcited by 55

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large Language Models (LLMs) are nowadays expected to generate content aligned with human preferences. Current work focuses on alignment at model training time, through techniques such as Reinforcement Learning with Human Feedback (RLHF). However, it is unclear if such methods are an effective choice to teach alignment objectives to the model. First, the inability to incorporate multiple, custom rewards and reliance on a model developer's view of universal and static principles are key limitations. Second, the reliability of such approaches is also questionable (e.g. susceptibility to jailbreaking even after safety training). To address these issues, we propose DeAL, a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view decoding as a heuristic-guided search process and facilitate the use of a wide variety of alignment objectives. Our experiments with programmatic constraints such as keyword and length constraints, and abstract objectives such as harmlessness and helpfulness, show that we can DeAL with fine-grained trade-offs and improve adherence to alignment objectives. Lastly, we demonstrate that DeAL is largely complementary to existing alignment strategies, and can be effectively paired with RLHF and prompting techniques to achieve better alignment.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 65 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

46 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  2. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  3. 2023Preference Ranking Optimization for Human Alignment30 Jun
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
30 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2018Hierarchical Neural Story Generationcited by 1,182
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  12. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  13. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  14. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  15. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  16. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 195
  17. 2023Efficient Guided Generation for Large Language Modelscited by 192
  18. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  19. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  20. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  21. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  22. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  23. 2016A Simple, Fast Diverse Decoding Algorithm for Neural Generationcited by 58
  24. 2023Grammar Prompting for Domain-Specific Language Generation with Large Language Modelscited by 58
  25. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49
  26. 2018Importance of Search and Evaluation Strategies in Neural Dialogue Modelingcited by 44
  27. 2020Best-First Beam Searchcited by 40
  28. 2023Faithfulness-Aware Decoding Strategies for Abstractive Summarizationcited by 31
  29. 2023Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsingcited by 6
  30. 2022Contrastive Training Improves Zero-Shot Classification of Semi-structured Documentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 21; 2026: 12.

55 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  3. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  4. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  5. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  6. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
  7. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  8. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Modelscited by 32
  9. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  10. 2024Evaluating the Prompt Steerability of Large Language Modelscited by 26
  11. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  12. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximizationcited by 20
38 more
  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  2. 2024Towards Inference-time Category-wise Safety Steering for Large Language Modelscited by 17
  3. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  4. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  5. 2025MetaScale: Test-Time Scaling with Evolving Meta-Thoughtscited by 13
  6. 2025Collab: Controlled Decoding using Mixture of Agents for LLM Alignmentcited by 13
  7. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  8. 2025On-the-fly Preference Alignment via Principle-Guided Decodingcited by 10
  9. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  10. 2024CBF-LLM: Safe Control for LLM Alignmentcited by 8
  11. 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Timecited by 8
  12. 2025Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Expertscited by 8
  13. 2025Inference-time Alignment in Continuous Spacecited by 7
  14. 2024Humanizing the Machine: Proxy Attacks to Mislead LLM Detectorscited by 6
  15. 2024SED: Self-Evaluation Decoding Enhances Large Language Models for Better Generationcited by 5
  16. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Marginscited by 5
  17. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5
  18. 2026ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generationcited by 5
  19. 2024Inference Time Alignment with Reward-Guided Tree Searchcited by 4
  20. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
  21. 2026nabla-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Spacecited by 4
  22. 2024Reranking Laws for Language Generation: A Communication-Theoretic Perspectivecited by 3
  23. 2025Watermarking Degrades Alignment in Language Models: Analysis and Mitigationcited by 3
  24. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  25. 2024Open-domain Implicit Format Control for Large Language Model Generationcited by 2
  26. 2024Inference time LLM alignment in single and multidomain preference spectrumcited by 2
  27. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  28. 2025Tokenized Bandit for LLM Decoding and Alignmentcited by 2
  29. 2025Precise Attribute Intensity Control in Large Language Models via Targeted Representation Editingcited by 2
  30. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspectivecited by 2
  31. 2026Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agentscited by 2
  32. 2026Activation Steering of Video Generation Models via Reduced-Order Linear Optimal Controlcited by 2
  33. 2025Saffron-1: Safety Inference Scalingcited by 1
  34. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1
  35. 2025Alignment-Aware Decodingcited by 1
  36. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  37. 2026ALIGNBEAM : Inference-Time Alignment Transfer via Cross-Vocabulary Logit Mixingcited by 1
  38. 2025Control Barrier Function for Aligning Large Language Modelscited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoding-Time Language Model Alignment with Multiple Objectives
  2. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  3. 2025A Survey on Training-free Alignment of Large Language Models
  4. 2024Decoding-time Realignment of Language Models
  5. 2023Large Language Model Alignment: A Survey
  6. 2023Aligning Large Language Models through Synthetic Feedback
  7. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
  8. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time