mlsanityData to 7 Oct 2026

Critic-Guided Decoding for Controlled Text Generation

arXiv 2212.10938 ↗First posted 21 Dec 2022cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

Steering language generation towards objectives or away from undesired content has been a long-standing goal in utilizing language models (LM). Recent work has demonstrated reinforcement learning and weighted decoding as effective approaches to achieve a higher level of language control and quality with pros and cons. In this work, we propose a novel critic decoding method for controlled language generation (CriticControl) that combines the strengths of reinforcement learning and weighted decoding. Specifically, we adopt the actor-critic framework to train an LM-steering critic from non-differentiable reward models. And similar to weighted decoding, our method freezes the language model and manipulates the output token distribution using called critic, improving training efficiency and stability. Evaluation of our method on three controlled generation tasks, namely topic control, sentiment control, and detoxification, shows that our approach generates more coherent and well-controlled texts than previous methods. In addition, CriticControl demonstrates superior generalization ability in zero-shot settings. Human evaluation studies also corroborate our findings.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 50 of its 68 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

50 of 68 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
38 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  3. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2018Hierarchical Neural Story Generationcited by 1,182
  6. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  7. 2018Neural Network Acceptability Judgmentscited by 952
  8. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  10. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  11. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  12. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  13. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  14. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  15. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  16. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  17. 2017Toward Controlled Generation of Textcited by 271
  18. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  19. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  20. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  21. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  22. 2018Learning to Write with Cooperative Discriminatorscited by 119
  23. 2017Deceiving Google's Perspective API Built for Detecting Toxic Commentscited by 99
  24. 2019Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfercited by 81
  25. 2021Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approachcited by 75
  26. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  27. 2022Controllable Natural Language Generation with Contrastive Prefixescited by 75
  28. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  29. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  30. 2022Mix and Match: Learning-free Controllable Text Generation using Energy Language Modelscited by 66
  31. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  32. 2022Controllable Text Generation with Neurally-Decomposed Oraclecited by 41
  33. 2019Reinforcement Learning Based Text Style Transfer without Parallel Training Corpuscited by 38
  34. 2022DIRECTOR: Generator-Classifiers For Supervised Language Modelingcited by 38
  35. 2019Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluatorscited by 34
  36. 2022A Distributional Lens for Multi-Aspect Controllable Text Generationcited by 33
  37. 2022Efficient Reinforcement Learning for Unsupervised Controlled Text Generationcited by 7
  38. 2020Discrete Action On-Policy Learning with Action-Value Criticcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 20; 2025: 9; 2026: 10.

45 papers.

Sort:

By area:

  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  3. 2023Controlled Decoding from Language Modelscited by 133
  4. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  5. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  6. 2023Controlled Text Generation via Language Model Arithmeticcited by 51
  7. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 37
  8. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Modelscited by 32
  9. 2024Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metriccited by 23
  10. 2024SafeInfer: Context Adaptive Decoding Time Safety Alignment for Large Language Modelscited by 22
  11. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decodingcited by 21
  12. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
33 more
  1. 2024Inference-Time Language Model Alignment via Integrated Value Guidancecited by 19
  2. 2024Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translationcited by 17
  3. 2024Controlled Text Generation for Large Language Model with Dynamic Attribute Graphscited by 16
  4. 2023LifeTox: Unveiling Implicit Toxicity in Life Advicecited by 14
  5. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferencescited by 13
  6. 2024AdvisorQA: Towards Helpful and Harmless Advice-seeking Question Answering with Collective Intelligencecited by 12
  7. 2023Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societiescited by 11
  8. 2024Potential and Challenges of Model Editing for Social Debiasingcited by 10
  9. 2024Controllable Text Generation in the Instruction-Tuning Eracited by 7
  10. 2024Large Language Models can be Strong Self-Detoxifierscited by 7
  11. 2024Guaranteed Generation from Large Language Modelscited by 7
  12. 2025Speculative Safety-Aware Decodingcited by 7
  13. 2024FreeCtrl: Constructing Control Centers with Feedforward Layers for Learning-Free Controllable Text Generationcited by 6
  14. 2024Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategiescited by 6
  15. 2025BiasFilter: An Inference-Time Debiasing Framework for Large Language Modelscited by 6
  16. 2025Evolutionary Guided Decoding: Iterative Value Refinement for LLMscited by 3
  17. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  18. 2024A Framework for Real-time Safeguarding the Text Generation of Large Language Modelcited by 2
  19. 2024LaSEr-Edit: Localized Span-level Error Editing with Energy-based Localizationcited by 2
  20. 2024Learning Goal-Conditioned Representations for Language Reward Modelscited by 2
  21. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  22. 2026Test-Time Detoxification without Training or Learning Anythingcited by 2
  23. 2026DLM-SWAI: Steering Diffusion Language Models Before They Unmaskcited by 2
  24. 2025Mimicking How Humans Interpret Out-of-Context Sentences Through Controlled Toxicity Decodingcited by 1
  25. 2026Limits of n-gram Style Control for LLMs via Logit-Space Injectioncited by 1
  26. 2025Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusingcited by 0
  27. 2025Test-time reward-guided alignment of language models by importance sampling on pre-logit spacecited by 0
  28. 2026OLLM: Options-based Large Language Modelscited by 0
  29. 2026Guidance Is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Modelscited by 0
  30. 2026A Comparative Study of Controlled Text Generation Systems Using Level-Playing-Field Evaluation Principlescited by 0
  31. 2026Latent Action Reparameterization for Efficient Agent Inferencecited by 0
  32. 2026Robust Critics: Defending LLMs Against Multi-Turn Attackscited by 0
  33. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  2. 2025Teaching Language Models to Critique via Reinforcement Learning
  3. 2024A Critical Look At Tokenwise Reward-Guided Text Generation
  4. 2026Adaptive Decoding via Test-Time Policy Learning for Self-Improving Generation
  5. 2024On the Low-Rank Parametrization of Reward Models for Controlled Language Generation
  6. 2025Text2Grad: Reinforcement Learning from Natural Language Feedback
  7. 2023The Consensus Game: Language Model Generation via Equilibrium Search
  8. 2025Towards Cost-Effective Reward Guided Text Generation