mlsanityData to 7 Oct 2026

Teacher Forcing Recovers Reward Functions for Text Generation

arXiv 2210.08708 ↗First posted 17 Oct 2022cs.LG cs.AI cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) has been widely used in text generation to alleviate the exposure bias issue or to utilize non-parallel datasets. The reward function plays an important role in making RL training successful. However, previous reward functions are typically task-specific and sparse, restricting the use of RL. In our work, we propose a task-agnostic approach that derives a step-wise reward function directly from a model trained with teacher forcing. We additionally propose a simple modification to stabilize the RL training on non-parallel datasets with our induced reward function. Empirical results show that our method outperforms self-training and reward regression methods on several text generation tasks, confirming the effectiveness of our reward function.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 42 of its 70 references that are in the map. First posted on arXiv 17 Oct 2022.

Builds on

42 of 70 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  10. 2016Sequence-Level Knowledge Distillationcited by 761
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
30 more
  1. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  2. 2017Convolutional Sequence to Sequence Learningcited by 550
  3. 2017Non-Autoregressive Neural Machine Translationcited by 457
  4. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  5. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  6. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 126
  7. 2015Minimum Risk Training for Neural Machine Translationcited by 114
  8. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  9. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93
  10. 2020Analyzing the Source and Target Contributions to Predictions in Neural Machine Translationcited by 71
  11. 2020Text Generation by Learning from Demonstrationscited by 64
  12. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  13. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  14. 2015Learning to Search Better Than Your Teachercited by 50
  15. 2018Toward Diverse Text Generation with Inverse Reinforcement Learningcited by 41
  16. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  17. 2019Unsupervised Paraphrasing by Simulated Annealingcited by 37
  18. 2020Unsupervised Text Generation by Learning from Searchcited by 36
  19. 2020DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterancescited by 36
  20. 2017Bandit Structured Prediction for Neural Sequence-to-Sequence Learningcited by 31
  21. 2021Diversifying Dialog Generation via Adaptive Label Smoothingcited by 25
  22. 2021Relating Neural Text Degeneration to Exposure Biascited by 19
  23. 2021Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translationcited by 17
  24. 2016Stochastic Structured Prediction under Bandit Feedbackcited by 12
  25. 2019An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generationcited by 11
  26. 2022An Equal-Size Hard EM Algorithm for Diverse Dialogue Generationcited by 11
  27. 2020The Gutenberg Dialogue Datasetcited by 8
  28. 2021Mixed Cross Entropy Loss for Neural Machine Translationcited by 8
  29. 2022An Empirical Study on the Overlapping Problem of Open-Domain Dialogue Datasetscited by 8
  30. 2021Learning to Selectively Learn for Weakly-supervised Paraphrase Generationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 4; 2025: 3; 2026: 1.

11 papers.

  1. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  2. 2024A Critical Look At Tokenwise Reward-Guided Text Generationcited by 12
  3. 2023Token Imbalance Adaptation for Radiology Report Generationcited by 9
  4. 2025Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Modelscited by 8
  5. 2022Inverse Reinforcement Learning for Text Summarizationcited by 5
  6. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learningcited by 5
  7. 2025Revisiting Intermediate-Layer Matching in Knowledge Distillation: Layer-Selection Strategy Doesn't Matter (Much)cited by 5
  8. 2024Adversarial Moment-Matching Distillation of Large Language Modelscited by 4
  9. 2024LLMR: Knowledge Distillation with a Large Language Model-Induced Rewardcited by 3
  10. 2025KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillationcited by 2
  11. 2026Multi-Step Semantic Reasoning in Generative Retrievalcited by 1