mlsanityData to 7 Oct 2026

Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback

arXiv 1707.07402 ↗First posted 24 Jul 2017cs.CL cs.AI cs.HCcited by 71

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve. Yet, current neural machine translation training focuses on expensive human-generated reference translations. We describe a reinforcement learning algorithm that improves neural machine translation systems from simulated human feedback. Our algorithm combines the advantage actor-critic algorithm (Mnih et al., 2016) with the attention-based neural encoder-decoder architecture (Luong et al., 2015). This algorithm (a) is well-designed for problems with a large action space and delayed rewards, (b) effectively optimizes traditional corpus-level machine translation metrics, and (c) is robust to skewed, high-variance, granular feedback modeled after actual human behaviors.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 47 references that are in the map. First posted on arXiv 24 Jul 2017.

Builds on

13 of 47 references.

References · 13

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  4. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  5. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  6. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  7. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  8. 2017Device Placement Optimization with Reinforcement Learningcited by 84
  9. 2015Learning to Search Better Than Your Teachercited by 50
  10. 2016Personalized Machine Translation: Preserving Original Author Traitscited by 47
  11. 2017Bandit Structured Prediction for Neural Sequence-to-Sequence Learningcited by 31
  12. 2016Stochastic Structured Prediction under Bandit Feedbackcited by 12
1 more
  1. 2016Bandit Structured Prediction for Learning from Partial Feedback in Statistical Machine Translationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 2; 2021: 7; 2022: 9; 2023: 22; 2024: 17; 2025: 10; 2026: 2.

71 papers.

Sort:

By area:

  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  7. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  8. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  9. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  10. 2024Critique-out-Loud Reward Modelscited by 105
  11. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  12. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
38 more
  1. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  2. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  3. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  4. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  5. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  6. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  7. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  8. 2023Learning to Generate Better Than Your LLMcited by 49
  9. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  10. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  11. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  12. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Modelcited by 33
  13. 2021Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 32
  14. 2021Interactive Learning from Activity Descriptioncited by 26
  15. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
  16. 2021Reliability Testing for Natural Language Processing Systemscited by 25
  17. 2024Aligning Target-Aware Molecule Diffusion Models with Exact Energy Optimizationcited by 25
  18. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  19. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  20. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  21. 2022Reward Modeling for Mitigating Toxicity in Transformer-based Language Modelscited by 17
  22. 2023Privately Aligning Language Models with Reinforcement Learningcited by 17
  23. 2023ESRL: Efficient Sampling-based Reinforcement Learning for Sequence Generationcited by 15
  24. 2021Diversity driven Query Rewriting in Search Advertisingcited by 14
  25. 2022Simulating Bandit Learning from User Feedback for Extractive Question Answeringcited by 14
  26. 2023Prompt-Based Length Controlled Generation with Reinforcement Learningcited by 14
  27. 2023STEER: Unified Style Transfer with Expert Reinforcementcited by 14
  28. 2020Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Taskscited by 12
  29. 2024Graph Diffusion Policy Optimizationcited by 12
  30. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  31. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  32. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  33. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  34. 2024MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learningcited by 10
  35. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translationcited by 8
  36. 2021Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effortcited by 7
  37. 2022MAD for Robust Reinforcement Learning in Machine Translationcited by 7
  38. 2023Advancements in Scientific Controllable Text Generation Methodscited by 7

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution
  2. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference
  3. 2026PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learning
  4. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning
  5. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation
  6. 2024Multi-perspective Alignment for Increasing Naturalness in Neural Machine Translation
  7. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
  8. 2020Learning Rewards from Linguistic Feedback