mlsanityData to 7 Oct 2026

On the Weaknesses of Reinforcement Learning for Neural Machine Translation

arXiv 1907.01752 ↗First posted 3 Jul 2019cs.CL cs.AI cs.LGcited by 82

Not on the map. Why?

Cited by 35 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning (RL) is frequently used to increase performance in text generation tasks, including machine translation (MT), notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). However, little is known about what and how these methods learn in the context of MT. We prove that one of the most common RL methods for MT does not optimize the expected reward, as well as show that other methods take an infeasibly long time to converge. In fact, our results suggest that RL practices in MT are likely to improve performance only where the pre-trained parameters are already close to yielding the correct translation. Our findings further suggest that observed gains may be due to effects unrelated to the training signal, but rather from changes in the shape of the distribution curve.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 11; 2022: 14; 2023: 12; 2024: 23; 2025: 17; 2026: 3.

82 papers.

Sort:

By area:

  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  3. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  6. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 199
  7. 2023Mitigating the Alignment Tax of RLHFcited by 181
  8. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  10. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  11. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  12. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
38 more
  1. 2024Learning Dynamics of LLM Finetuningcited by 93
  2. 2021Survey of Low-Resource Machine Translationcited by 92
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  4. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  5. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  6. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  7. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  8. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  9. 2020Text Generation by Learning from Demonstrationscited by 64
  10. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
  11. 2021Machine Translation Decoding beyond Beam Searchcited by 55
  12. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 48
  13. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
  14. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  15. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  16. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  17. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 32
  18. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 28
  19. 2023Tailoring Language Generation Models under Total Variation Distancecited by 27
  20. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
  21. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  22. 2025Improving LLM General Preference Alignment via Optimistic Online Mirror Descentcited by 24
  23. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
  24. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
  25. 2021Boosting Search Engines with Interactive Agentscited by 19
  26. 2022Why is constrained neural language generation particularly challenging?cited by 19
  27. 2023ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generationcited by 15
  28. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  29. 2020Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Taskscited by 12
  30. 2021Reward Optimization for Neural Machine Translation with Learned Metricscited by 12
  31. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  32. 2024Towards Federated RLHF with Aggregated Client Preference for LLMscited by 11
  33. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowdcited by 11
  34. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  35. 2023SuperHF: Supervised Iterative Learning from Human Feedbackcited by 10
  36. 2024YODA: Teacher-Student Progressive Learning for Language Modelscited by 10
  37. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignmentcited by 9
  38. 2024Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMscited by 9

50 most cited of 82 All 82 (JSON)