mlsanityData to 7 Oct 2026

Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks

arXiv 2011.02511 ↗First posted 4 Nov 2020cs.CL cs.LGcited by 12

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Large volumes of interaction logs can be collected from NLP systems that are deployed in the real world. How can this wealth of information be leveraged? Using such interaction logs in an offline reinforcement learning (RL) setting is a promising approach. However, due to the nature of NLP tasks and the constraints of production systems, a series of challenges arise. We present a concise overview of these challenges and discuss possible solutions.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 50 references that are in the map. First posted on arXiv 4 Nov 2020.

Builds on

25 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  2. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning27 May
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 22

  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  3. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  4. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  5. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  6. 2019Challenges of Real-World Reinforcement Learningcited by 282
  7. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  8. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  9. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  10. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  11. 2018Reinforcement Learning with Perturbed Rewardscited by 76
  12. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
10 more
  1. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  2. 2017Task-Oriented Query Reformulation with Reinforcement Learningcited by 39
  3. 2019Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialogcited by 37
  4. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32
  5. 2017End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradientcited by 23
  6. 2017Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translationcited by 13
  7. 2016Stochastic Structured Prediction under Bandit Feedbackcited by 12
  8. 2017Batch Policy Gradient Methods for Improving Neural Conversation Modelscited by 10
  9. 2018A Reinforcement Learning Approach to Interactive-Predictive Neural Machine Translationcited by 8
  10. 2017Counterfactual Learning for Machine Translation: Degeneracies and Solutionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 3; 2024: 3; 2025: 2; 2026: 1.

12 papers.

  1. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  2. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  3. 2024Participation in the age of foundation modelscited by 32
  4. 2024Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversationscited by 13
  5. 2022Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarizationcited by 12
  6. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  7. 2025Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generationcited by 10
  8. 2023Simple Embodied Language Learning as a Byproduct of Meta-Reinforcement Learningcited by 3
  9. 2023SPEECH: Structured Prediction with Energy-Based Event-Centric Hyperspherescited by 2
  10. 2024TCR-GPT: Integrating Autoregressive Model and Reinforcement Learning for T-Cell Receptor Repertoires Generationcited by 0
  11. 2025PARL: Prompt-based Agents for Reinforcement Learningcited by 0
  12. 2026From Automation to Collaboration: Human-in-the-Loop Methods for Safe and Trustworthy NLPcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  2. 2022Leveraging Offline Data in Online Reinforcement Learning
  3. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  4. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedback
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  7. 2023The Generalization Gap in Offline Reinforcement Learning
  8. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog