mlsanityData to 7 Oct 2026

Reinforcement Learning with Token-level Feedback for Controllable Text Generation

arXiv 2403.11558 ↗First posted 18 Mar 2024cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

To meet the requirements of real-world applications, it is essential to control generations of large language models (LLMs). Prior research has tried to introduce reinforcement learning (RL) into controllable text generation while most existing methods suffer from overfitting issues (finetuning-based methods) or semantic collapse (post-processing methods). However, current RL methods are generally guided by coarse-grained (sentence/paragraph-level) feedback, which may lead to suboptimal performance owing to semantic twists or progressions within sentences. To tackle that, we propose a novel reinforcement learning algorithm named TOLE which formulates TOken-LEvel rewards for controllable text generation, and employs a "first-quantize-then-noise" paradigm to enhance the robustness of the RL algorithm.Furthermore, TOLE can be flexibly extended to multiple constraints with little computational expense. Experimental results show that our algorithm can achieve superior performance on both single-attribute and multi-attribute control tasks. We have released our codes at https://github.com/WindyLee0822/CTG

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

30 of 35 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  10. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  11. 2021Ethical and social risks of harm from Language Modelscited by 957
  12. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
18 more
  1. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  3. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  5. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  6. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  7. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  8. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 147
  9. 2020A Distributional Approach to Controlled Text Generationcited by 117
  10. 2022Controllable Natural Language Generation with Contrastive Prefixescited by 75
  11. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  12. 2022A Distributional Lens for Multi-Aspect Controllable Text Generationcited by 33
  13. 2023Hyper-Decision Transformer for Efficient Online Policy Adaptationcited by 32
  14. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  15. 2022DisCup: Discriminator Cooperative Unlikelihood Prompt-tuning for Controllable Text Generationcited by 30
  16. 2022Controllable Text Generation via Probability Density Estimation in the Latent Spacecited by 27
  17. 2021Plug-and-Blend: A Framework for Controllable Story Generation with Blended Control Codescited by 19
  18. 2022An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 24
  2. 2025Kardia-R1: Unleashing LLMs to Reason toward Understanding and Empathy for Emotional Support via Rubric-as-Judge Reinforcement Learningcited by 22
  3. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillationcited by 15
  4. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  5. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  6. 2024ReflectDiffu:Reflect between Emotion-intent Contagion and Mimicry for Empathetic Response Generation via a RL-Diffusion Frameworkcited by 6
  7. 2025JAM: Controllable and Responsible Text Generation via Causal Reasoning and Latent Vector Manipulationcited by 5
  8. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  9. 2025Quality-Aware Decoding: Unifying Quality Estimation and Decodingcited by 1
  10. 2025LaPA2: Length-Aware Prefix and Prompt Attention Augmentation for Long-Form Controllable Text Generationcited by 1
  11. 2026Position: AI Safety Requires Effective Controllabilitycited by 1
  12. 2026SWIM: Student Writing Simulation via Proficiency-Conditioned Generationcited by 1
4 more
  1. 2024Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehensioncited by 0
  2. 2025RL-MoE: An Image-Based Privacy Preserving Approach In Intelligent Transportation Systemcited by 0
  3. 2026HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learningcited by 0
  4. 2026The Illusion of Control: Why Bare Classifier Inversion Silently Fails in Concept-Bottleneck Text Generationcited by 0