mlsanityData to 7 Oct 2026

Efficient (Soft) Q-Learning for Text Generation with Limited Good Data

arXiv 2106.07704 ↗First posted 14 Jun 2021cs.CL cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Maximum likelihood estimation (MLE) is the predominant algorithm for training text generation models. This paradigm relies on direct supervision examples, which is not applicable to many emerging applications, such as generating adversarial attacks or generating prompts to control language models. Reinforcement learning (RL) on the other hand offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. Yet previous RL algorithms for text generation, such as policy gradient (on-policy RL) and Q-learning (off-policy RL), are often notoriously inefficient or unstable to train due to the large sequence space and the sparse reward received only at the end of sequences. In this paper, we introduce a new RL formulation for text generation from the soft Q-learning (SQL) perspective. It enables us to draw from the latest RL advances, such as path consistency learning, to combine the best of on-/off-policy updates, and learn effectively from sparse reward. We apply the approach to a wide range of novel text generation tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation. Experiments show our approach consistently outperforms both task-specialized algorithms and the previous RL methods.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 54 of its 73 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

54 of 73 references.

References · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  10. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  11. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
38 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  4. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  5. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  7. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  8. 2016Self-critical Sequence Training for Image Captioningcited by 544
  9. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  10. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  11. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  12. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  13. 2017Toward Controlled Generation of Textcited by 271
  14. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  15. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  16. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  17. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  18. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  19. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  20. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  21. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  22. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  23. 2020Towards Controllable Biases in Language Generationcited by 112
  24. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  25. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  26. 2018Language GANs Falling Shortcited by 97
  27. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  28. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  29. 2016Reward Augmented Maximum Likelihood for Neural Structured Predictioncited by 80
  30. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  31. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  32. 2015Language Understanding for Text-based Games Using Deep Reinforcement Learningcited by 70
  33. 2020Text Generation by Learning from Demonstrationscited by 64
  34. 2020Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generationcited by 47
  35. 2018Toward Diverse Text Generation with Inverse Reinforcement Learningcited by 41
  36. 2017Trust-PCL: An Off-Policy Trust Region Method for Continuous Controlcited by 29
  37. 2020Generating Label Cohesive and Well-Formed Adversarial Claimscited by 27
  38. 2017End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradientcited by 23

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 16; 2025: 9; 2026: 1.

35 papers.

Sort:

By area:

  1. 2023Amortizing intractable inference in large language modelscited by 94
  2. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  3. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  4. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  5. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  6. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  7. 2024PRewrite: Prompt Rewriting with Reinforcement Learningcited by 43
  8. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  9. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  10. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  11. 2024On the Privacy Risk of In-context Learningcited by 31
  12. 2025InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignmentcited by 18
23 more
  1. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
  2. 2024MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimizationcited by 16
  3. 2023GROVE: A Retrieval-augmented Complex Story Generation Framework with A Forest of Evidencecited by 14
  4. 2024Learning to Plan for Language Modeling from Unlabeled Datacited by 12
  5. 2024Can a Bayesian Oracle Prevent Harm from an Agent?cited by 12
  6. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  7. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  8. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  9. 2025Self-Guided Process Reward Optimization with Redefined Step-wise Advantage for Process Reinforcement Learningcited by 10
  10. 2024Contrastive Learning with Counterfactual Explanations for Radiology Report Generationcited by 8
  11. 2024M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Modelscited by 7
  12. 2023PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Trainingcited by 6
  13. 2025Supervised Optimism Correction: Be Confident When LLMs Are Surecited by 5
  14. 2025Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignmentcited by 5
  15. 2025How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptationcited by 4
  16. 2026On the Power of (Approximate) Reward Models for Inference-Time Scalingcited by 4
  17. 2025ShiQ: Bringing back Bellman to LLMscited by 3
  18. 2023The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challengescited by 2
  19. 2024RIFF: Learning to Rephrase Inputs for Few-shot Fine-tuning of Language Modelscited by 2
  20. 2024Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?cited by 1
  21. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  22. 2024APrompt4EM: Augmented Prompt Tuning for Generalized Entity Matchingcited by 0
  23. 2024An Automatic Prompt Generation System for Tabular Data Taskscited by 0