mlsanityData to 14 Oct 2026

Alignment-Aware Decoding

arXiv 2509.26169 ↗First posted 30 Sep 2025cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment of large language models remains a central challenge in natural language processing. Preference optimization has emerged as a popular and effective method for improving alignment, typically through training-time or prompt-based interventions. In this paper, we introduce alignment-aware decoding (AAD), a method to enhance model alignment directly at inference. Theoretically, AAD can be interpreted as implicit reward optimization, yet it requires no specialized training beyond the standard DPO setup. Empirically, AAD consistently outperforms strong baselines across diverse alignment benchmarks and model scales. Moreover, in data-constrained settings, AAD can produce high-quality synthetic data to improve alignment under standard decoding, providing a practical solution when labeled data is limited.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 52 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

40 of 52 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Inference-time Alignment in Continuous Space26 May
  2. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model24 Oct
  3. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment10 Oct
  4. 2024Inference-Time Language Model Alignment via Integrated Value Guidance26 Sep
  5. 2024Selective Preference Optimization via Token-Level Reward Function Estimation24 Aug
  6. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models29 May
  7. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  8. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  9. 2024DeAL: Decoding-time Alignment for Large Language Models5 Feb
  10. 2024Decoding-time Realignment of Language Models5 Feb
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment4 Jun
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2016Concrete Problems in AI Safetycited by 1,642
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  10. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  11. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
15 more
  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  3. 2024Iterative Reasoning Preference Optimizationcited by 246
  4. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  5. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  6. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  7. 2023Controlled Decoding from Language Modelscited by 135
  8. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  9. 2024Tuning Language Models by Proxycited by 89
  10. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  11. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  12. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 60
  13. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  14. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  15. 2024Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Modelcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Convex Optimization for Alignment and Preference Learning on a Single GPUcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Nudging: Inference-time Alignment of LLMs via Guided Decoding
  2. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  3. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decoding
  4. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
  5. 2026Safe Inference-Time Alignment via Lagrangian Reward Augmentation
  6. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignment
  7. 2024InfAlign: Inference-aware language model alignment
  8. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models