mlsanityData to 14 Oct 2026

Decoding-time Realignment of Language Models

arXiv 2402.02992 ↗First posted 5 Feb 2024cs.LG cs.AI cs.CLcited by 67

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning language models with human preferences is crucial for reducing errors and biases in these models. Alignment techniques, such as reinforcement learning from human feedback (RLHF), are typically cast as optimizing a tradeoff between human preference rewards and a proximity regularization term that encourages staying close to the unaligned model. Selecting an appropriate level of regularization is critical: insufficient regularization can lead to reduced model capabilities due to reward hacking, whereas excessive regularization hinders alignment. Traditional methods for finding the optimal regularization level require retraining multiple models with varying regularization strengths. This process, however, is resource-intensive, especially for large models. To address this challenge, we propose decoding-time realignment (DeRa), a simple method to explore and evaluate different regularization strengths in aligned models without retraining. DeRa enables control over the degree of alignment, allowing users to smoothly transition between unaligned and aligned models. It also enhances the efficiency of hyperparameter tuning by enabling the identification of effective regularization strengths using a validation dataset.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 46 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

40 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2017Deep reinforcement learning from human preferencescited by 3,912
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023Mistral 7Bcited by 3,164
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
24 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2016Concrete Problems in AI Safetycited by 1,642
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  10. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  11. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  12. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  13. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  14. 2023Nash Learning from Human Feedbackcited by 224
  15. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  16. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  17. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  18. 2023Stay on topic with Classifier-Free Guidancecited by 91
  19. 2024Tuning Language Models by Proxycited by 89
  20. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  21. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  22. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  23. 2018Simple Fusion: Return of the Language Modelcited by 30
  24. 2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 25; 2025: 28; 2026: 14.

67 papers.

Sort:

By area:

  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  3. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  4. 2025Harnessing Multiple Large Language Models: A Survey on LLM Ensemblecited by 86
  5. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  6. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  7. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 48
  8. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  9. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 43
  10. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 39
  11. 2024Model Extrapolation Expedites Alignmentcited by 37
  12. 2024Personality Alignment of Large Language Modelscited by 36
38 more
  1. 2024Erasing Conceptual Knowledge from Language Modelscited by 34
  2. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  3. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignmentcited by 27
  4. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 27
  5. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 27
  6. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  7. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  8. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Modelcited by 22
  9. 2025Robust Multi-Objective Preference Alignment with Online DPOcited by 22
  10. 2025PILAF: Optimal Human Preference Sampling for Reward Modelingcited by 20
  11. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  12. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  13. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 13
  14. 2025Fast Large Language Model Collaborative Decoding via Speculationcited by 12
  15. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  16. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
  17. 2025ProxyThinker: Test-Time Guidance through Small Visual Reasonerscited by 8
  18. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  19. 2024Large Language Models can be Strong Self-Detoxifierscited by 7
  20. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 7
  21. 2025Inference-time Alignment in Continuous Spacecited by 7
  22. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 7
  23. 2024Humanizing the Machine: Proxy Attacks to Mislead LLM Detectorscited by 6
  24. 2025Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignmentcited by 6
  25. 2024FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devicescited by 5
  26. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  27. 2025KL Penalty Control via Perturbation for Direct Preference Optimizationcited by 5
  28. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  29. 2024Reranking Laws for Language Generation: A Communication-Theoretic Perspectivecited by 4
  30. 2026EXACT: Explicit Attribute-Guided Decoding-Time Personalizationcited by 4
  31. 2025Watermarking Degrades Alignment in Language Models: Analysis and Mitigationcited by 3
  32. 2025Flexible Realignment of Language Modelscited by 3
  33. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
  34. 2026PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decodingcited by 3
  35. 2024Inference time LLM alignment in single and multidomain preference spectrumcited by 2
  36. 2025Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Modelscited by 2
  37. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  38. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DeAL: Decoding-time Alignment for Large Language Models
  2. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization
  3. 2025A Principled Loss Function for Direct Language Model Alignment
  4. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  5. 2024Towards Efficient Exact Optimization of Language Model Alignment
  6. 2023Aligning Language Models with Offline Learning from Human Feedback
  7. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  8. 2025Alignment is Localized: A Causal Probe into Preference Layers