mlsanityData to 7 Oct 2026

Controlled Decoding from Language Models

arXiv 2310.17022 ↗First posted 25 Oct 2023cs.LG cs.AI cs.CLcited by 133

Not on the map. Why?

Cited by 65 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

KL-regularized reinforcement learning (RL) is a popular alignment framework to control the language model responses towards high reward outcomes. We pose a tokenwise RL objective and propose a modular solver for it, called controlled decoding (CD). CD exerts control through a separate prefix scorer module, which is trained to learn a value function for the reward. The prefix scorer is used at inference time to control the generation from a frozen base model, provably sampling from a solution to the RL objective. We empirically demonstrate that CD is effective as a control mechanism on popular benchmarks. We also show that prefix scorers for multiple rewards may be combined at inference time, effectively solving a multi-objective RL problem with no additional training. We show that the benefits of applying CD transfer to an unseen base model with no further tuning as well. Finally, we show that CD can be applied in a blockwise decoding fashion at inference-time, essentially bridging the gap between the popular best-of-K strategy and tokenwise control through reinforcement learning. This makes CD a promising approach for alignment of language models.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

36 of 45 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
24 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  4. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  7. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  8. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  9. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  10. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  11. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  12. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  13. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  14. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  15. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  16. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  17. 2022Controllable Text Generation with Neurally-Decomposed Oraclecited by 41
  18. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  19. 2022DIRECTOR: Generator-Classifiers For Supervised Language Modelingcited by 38
  20. 2021PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decodingcited by 32
  21. 2022DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generationcited by 30
  22. 2017Learning to Decode for Future Successcited by 16
  23. 2021To Beam Or Not To Beam: That is a Question of Cooperation for Language GANscited by 14
  24. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 47; 2025: 54; 2026: 32.

133 papers.

Sort:

By area:

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  2. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  3. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  4. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  5. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  6. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  7. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  8. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  9. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  10. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  11. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  12. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 82
38 more
  1. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  2. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  3. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  4. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  5. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  6. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  7. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  8. 2024Variational Best-of-N Alignmentcited by 49
  9. 2024Asymptotics of Language Model Alignmentcited by 46
  10. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
  11. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  12. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  13. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  14. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  15. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 36
  16. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  17. 2024InfAlign: Inference-aware language model alignmentcited by 34
  18. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Explorationcited by 33
  19. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Modelscited by 32
  20. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  21. 2025Language Model Personalization via Reward Factorizationcited by 27
  22. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
  23. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  24. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  25. 2025Antidistillation Samplingcited by 24
  26. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Modelscited by 22
  27. 2024Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlocited by 21
  28. 2025SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Modelscited by 19
  29. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  30. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  31. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  32. 2025CoDe: Blockwise Control for Denoising Diffusion Modelscited by 14
  33. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  34. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  35. 2024Learning to Plan for Language Modeling from Unlabeled Datacited by 12
  36. 2024A Critical Look At Tokenwise Reward-Guided Text Generationcited by 12
  37. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 12
  38. 2024Tandem Transformers for Inference Efficient LLMscited by 10

50 most cited of 133 All 133 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Collab: Controlled Decoding using Mixture of Agents for LLM Alignment
  2. 2025A Theoretical Lens for RL-Tuned Language Models via Energy-Based Models
  3. 2026Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation
  4. 2023Language Model Alignment with Elastic Reset
  5. 2026From Base Rollouts to RL Reasoning: A Budgeted Search Perspective
  6. 2024DeAL: Decoding-time Alignment for Large Language Models
  7. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  8. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization