mlsanityData to 7 Oct 2026

Text Generation by Learning from Demonstrations

arXiv 2009.07839 ↗First posted 16 Sep 2020cs.CL cs.LGcited by 64

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation. This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objective and evaluation metric (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated). To alleviate these problems, we frame text generation as an offline reinforcement learning (RL) problem with expert demonstrations (i.e., the reference), where the goal is to maximize quality given model-generated histories. We propose GOLD (generation by off-policy learning from demonstrations): an easy-to-optimize algorithm that learns from the demonstrations by importance weighting. Intuitively, GOLD upweights confident tokens and downweights unconfident ones in the reference during training, avoiding optimization issues faced by prior RL approaches that rely on online data collection. According to both automatic and human evaluation, models trained by GOLD outperform those trained by MLE and policy gradient on summarization, question generation, and machine translation. Further, our models are less sensitive to decoding algorithms and alleviate exposure bias.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 66 references that are in the map. First posted on arXiv 16 Sep 2020.

Builds on

47 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog30 Jun

Other references · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  11. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  12. 2015Teaching Machines to Read and Comprehendcited by 1,039
34 more
  1. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  2. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  4. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  5. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  6. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  7. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  8. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  9. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  10. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  11. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  12. 2019On NMT Search Errors and Model Errors: Cat Got Your Tongue?cited by 114
  13. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
  14. 2018Language GANs Falling Shortcited by 97
  15. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  16. 2017Neural Question Generation from Text: A Preliminary Studycited by 78
  17. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 76
  18. 2020Improved Natural Language Generation via Loss Truncationcited by 75
  19. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  20. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  21. 2019Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generationcited by 62
  22. 2015Structured Prediction Energy Networkscited by 59
  23. 2017Paraphrase Generation with Deep Reinforcement Learningcited by 58
  24. 2020Consistency of a Recurrent Language Model With Respect to Incomplete Decodingcited by 54
  25. 2018Deep Reinforcement Learning For Sequence to Sequence Modelscited by 39
  26. 2017A Deep Reinforcement Learning Chatbotcited by 36
  27. 2019Mixture Content Selection for Diverse Sequence Generationcited by 27
  28. 2016Globally Normalized Transition-Based Neural Networkscited by 26
  29. 2017SEARNN: Training RNNs with Global-Local Lossescited by 17
  30. 2019A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Modelscited by 17
  31. 2018Split and Rephrase: Better Evaluation and a Stronger Baselinecited by 15
  32. 2017A Continuous Relaxation of Beam Search for End-to-end Training of Neural Sequence Modelscited by 10
  33. 2019Improving Joint Training of Inference Networks and Structured Prediction Energy Networkscited by 6
  34. 2018Evaluating Text GANs as Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 13; 2023: 32; 2024: 9; 2025: 4; 2026: 1.

64 papers.

Sort:

By area:

  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  3. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  4. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  5. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  6. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  7. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  8. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  9. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  10. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  11. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  12. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
38 more
  1. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  2. 2023Learning to Generate Better Than Your LLMcited by 49
  3. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  4. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  5. 2022Reward Gaming in Conditional Text Generationcited by 29
  6. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  7. 2023Tailoring Language Generation Models under Total Variation Distancecited by 27
  8. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  9. 2022VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignmentcited by 20
  10. 2024Prompt-Based Length Controlled Generation with Multiple Control Typescited by 19
  11. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
  12. 2023On the Efficacy of Sampling Adapterscited by 17
  13. 2023MixCE: Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropiescited by 16
  14. 2023ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generationcited by 15
  15. 2023Prompt-Based Length Controlled Generation with Reinforcement Learningcited by 14
  16. 2023Building Persona Consistent Dialogue Agents with Offline Reinforcement Learningcited by 13
  17. 2022Knowledge Infused Decodingcited by 12
  18. 2023Semi-Offline Reinforcement Learning for Optimized Text Generationcited by 12
  19. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11
  20. 2023Zero-shot Visual Question Answering with Language Model Feedbackcited by 11
  21. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  22. 2021AgreeSum: Agreement-Oriented Multi-Document Summarizationcited by 10
  23. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  24. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 10
  25. 2022GoSum: Extractive Summarization of Long Documents by Reinforcement Learning and Graph Organized discourse statecited by 9
  26. 2022KRLS: Improving End-to-End Response Generation in Task Oriented Dialog with Reinforced Keywords Learningcited by 9
  27. 2023Language Model Decoding as Direct Metrics Optimizationcited by 8
  28. 2024Language Generation with Strictly Proper Scoring Rulescited by 8
  29. 2021Language Model Augmented Relevance Scorecited by 7
  30. 2022MAD for Robust Reinforcement Learning in Machine Translationcited by 7
  31. 2023EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modelingcited by 7
  32. 2023On the Effectiveness of Offline RL for Dialogue Response Generationcited by 6
  33. 2023Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Modelscited by 6
  34. 2024Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understandingcited by 6
  35. 2021Amortized Noisy Channel Neural Machine Translationcited by 4
  36. 2023Beyond MLE: Convex Learning for Text Generationcited by 4
  37. 2024Adversarial Moment-Matching Distillation of Large Language Modelscited by 4
  38. 2026Diversity in Large Language Models under Supervised Fine-Tuningcited by 4

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2018Toward Diverse Text Generation with Inverse Reinforcement Learning
  2. 2025LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning
  3. 2025Text Generation Beyond Discrete Token Sampling
  4. 2025Towards Cost-Effective Reward Guided Text Generation
  5. 2022Quark: Controllable Text Generation with Reinforced Unlearning
  6. 2019Non-Monotonic Sequential Text Generation
  7. 2024A Critical Look At Tokenwise Reward-Guided Text Generation
  8. 2022Critic-Guided Decoding for Controlled Text Generation