mlsanityData to 7 Oct 2026

An Actor-Critic Algorithm for Sequence Prediction

arXiv 1607.07086 ↗First posted 24 Jul 2016cs.LGcited by 158

Not on the map. Why?

Cited by 27 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present an approach to training neural networks to generate sequences using actor-critic methods from reinforcement learning (RL). Current log-likelihood training methods are limited by the discrepancy between their training and testing modes, as models must generate tokens conditioned on their previous guesses rather than the ground-truth tokens. We address this problem by introducing a textitcritic network that is trained to predict the value of an output token, given the policy of an textitactor network. This results in a training procedure that is much closer to the test phase, and allows us to directly optimize for a task-specific score such as BLEU. Crucially, since we leverage these techniques in the supervised learning setting rather than the traditional RL setting, we condition the critic network on the ground-truth output. We show that our method leads to improved performance on both a synthetic task, and for German-English machine translation. Our analysis paves the way for such methods to be applied in natural language generation tasks, such as machine translation, caption generation, and dialogue modelling.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2017: 1; 2018: 2; 2019: 3; 2020: 4; 2021: 36; 2022: 23; 2023: 37; 2024: 33; 2025: 13; 2026: 6.

158 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  4. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  5. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  6. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  7. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  8. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  9. 2024The pitfalls of next-token predictioncited by 156
  10. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  11. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  12. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
38 more
  1. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  2. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  3. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  5. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  6. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  7. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  8. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  9. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  10. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  11. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  12. 2025Generalized Interpolating Discrete Diffusioncited by 68
  13. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  14. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  15. 2023A Study on ReLU and Softmax in Transformercited by 53
  16. 2021Recall@k Surrogate Loss with Large Batches and Similarity Mixupcited by 49
  17. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  18. 2021Deep Multiagent Reinforcement Learning: Challenges and Directionscited by 42
  19. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
  20. 2021Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithmcited by 39
  21. 2021Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphscited by 39
  22. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  23. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 32
  24. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  25. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithmcited by 26
  26. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
  27. 2025Audio Large Language Models Can Be Descriptive Speech Quality Evaluatorscited by 25
  28. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generationcited by 22
  29. 2021Creativity and Machine Learning: A Surveycited by 21
  30. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 21
  31. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  32. 2021AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Taskscited by 19
  33. 2024Prompt-Based Length Controlled Generation with Multiple Control Typescited by 19
  34. 2021Sequence-Level Training for Non-Autoregressive Neural Machine Translationcited by 18
  35. 2024PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preferencecited by 17
  36. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  37. 2021ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Modelscited by 16
  38. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16

50 most cited of 158 All 158 (JSON)