mlsanityData to 14 Oct 2026

Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning

arXiv 1805.10627 ↗First posted 27 May 2018cs.CL stat.MLcited by 57

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT). We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator, and the effect of the quality of reward estimates on the overall RL task. Our analysis of cardinal (5-point ratings) and ordinal (pairwise preferences) feedback shows that their intra- and inter-annotator α-agreement is comparable. Best reliability is obtained for standardized cardinal feedback, and cardinal feedback is also easiest to learn and generalize from. Finally, improvements of over 1 BLEU can be obtained by integrating a regression-based reward estimator trained on cardinal feedback for 800 translations into RL for NMT. This shows that RL is possible even from small amounts of fairly reliable human feedback, pointing to a great potential for applications at larger scale.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 63 references that are in the map. First posted on arXiv 27 May 2018.

Builds on

24 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  8. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  9. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  10. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  11. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  12. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
11 more
  1. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 159
  2. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  3. 2015Minimum Risk Training for Neural Machine Translationcited by 115
  4. 2016Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervisioncited by 103
  5. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 72
  6. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 72
  7. 2017Bandit Structured Prediction for Neural Sequence-to-Sequence Learningcited by 31
  8. 2016Fast Domain Adaptation for Neural Machine Translationcited by 28
  9. 2017Adversarial Neural Machine Translationcited by 21
  10. 2017Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translationcited by 13
  11. 2016Stochastic Structured Prediction under Bandit Feedbackcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 1; 2021: 4; 2022: 4; 2023: 16; 2024: 18; 2025: 10; 2026: 2.

57 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  4. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  5. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  7. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  8. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  9. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 54
  10. 2024Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Treescited by 38
  11. 2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
  12. 2024Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoningcited by 35
38 more
  1. 2024Suri: Multi-constraint Instruction Following for Long-form Text Generationcited by 34
  2. 2024Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Modelcited by 33
  3. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Modelscited by 32
  4. 2023Aligning Language Models with Human Preferences via a Bayesian Approachcited by 27
  5. 2021Interactive Learning from Activity Descriptioncited by 26
  6. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  7. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 23
  8. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 22
  9. 2023AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMscited by 22
  10. 2024Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedbackcited by 20
  11. 2024Your Weak LLM is Secretly a Strong Teacher for Alignmentcited by 20
  12. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  13. 2022Why is constrained neural language generation particularly challenging?cited by 19
  14. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inferencecited by 19
  15. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  16. 2024SmartRAG: Jointly Learn RAG-Related Tasks From the Environment Feedbackcited by 19
  17. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16
  18. 2024Improving Socratic Question Generation using Data Augmentation and Preference Optimizationcited by 16
  19. 2022Simulating Bandit Learning from User Feedback for Extractive Question Answeringcited by 14
  20. 2023Harnessing the Power of LLMs: Evaluating Human-AI Text Co-Creation through the Lens of News Headline Generationcited by 14
  21. 2020Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Taskscited by 12
  22. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  23. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  24. 2023Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modellingcited by 10
  25. 2023DRL4Route: A Deep Reinforcement Learning Framework for Pick-up and Delivery Route Predictioncited by 10
  26. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignmentcited by 10
  27. 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behaviorcited by 9
  28. 2026GDRO: Group-level Reward Post-training Suitable for Diffusion Modelscited by 8
  29. 2023Automatic Pair Construction for Contrastive Post-trainingcited by 7
  30. 2025ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimizationcited by 7
  31. 2025Disentangling Length Bias In Preference Learning Via Response-Conditioned Modelingcited by 6
  32. 2025Aligning Protein Conformation Ensemble Generation with Physical Feedbackcited by 5
  33. 2023Preference as Reward, Maximum Preference Optimization with Importance Samplingcited by 4
  34. 2024SumRecom: A Personalized Summarization Approach by Learning from Users' Feedbackcited by 4
  35. 2023LitSumm: Large language models for literature summarisation of non-coding RNAscited by 3
  36. 2024The Life Cycle of Large Language Models: A Review of Biases in Educationcited by 3
  37. 2024Post-edits Are Preferences Toocited by 3
  38. 2025SCRAMBLe : Enhancing Multimodal LLM Compositionality with Synthetic Preference Datacited by 2

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  2. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  3. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
  4. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards
  5. 2025Establishing Reliability Metrics for Reward Models in Large Language Models
  6. 2024Optimal Design for Reward Modeling in RLHF
  7. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Models
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint