mlsanityData to 7 Oct 2026

Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback

arXiv 2208.03270 ↗First posted 5 Aug 2022cs.CL cs.AIcited by 35

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Frozen models trained to mimic static datasets can never improve their performance. Models that can employ internet-retrieval for up-to-date information and obtain feedback from humans during deployment provide the promise of both adapting to new information, and improving their performance. In this work we study how to improve internet-driven conversational skills in such a learning framework. We collect deployment data, which we make publicly available, of human interactions, and collect various types of human feedback -- including binary quality measurements, free-form text feedback, and fine-grained reasons for failure. We then study various algorithms for improving from such feedback, including standard supervised learning, rejection sampling, model-guiding and reward-based learning, in order to make recommendations on which type of feedback and algorithms work best. We find the recently introduced Director model (Arora et al., '22) shows significant improvements over other existing approaches.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 37 references that are in the map. First posted on arXiv 5 Aug 2022.

Builds on

26 of 37 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  9. 2020Recipes for building an open-domain chatbotcited by 593
  10. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  11. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  12. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
14 more
  1. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  2. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  3. 2020The Pushshift Reddit Datasetcited by 289
  4. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  5. 2021Internet-Augmented Dialogue Generationcited by 217
  6. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
  7. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  8. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  9. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  10. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  11. 2022DIRECTOR: Generator-Classifiers For Supervised Language Modelingcited by 38
  12. 2021Reason first, then respond: Modular Generation for Knowledge-infused Dialoguecited by 32
  13. 2021Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identitycited by 22
  14. 2020Deploying Lifelong Open-Domain Dialogue Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 18; 2024: 9; 2025: 1; 2026: 1.

35 papers.

Sort:

By area:

  1. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  2. 2023Augmented Language Models: a Surveycited by 398
  3. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  4. 2023Training Language Models with Language Feedback at Scalecited by 125
  5. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  7. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  8. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  9. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  10. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  11. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  12. 2022The CRINGE Loss: Learning what language not to modelcited by 39
23 more
  1. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  2. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  3. 2022I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitationcited by 21
  4. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labelscited by 20
  5. 2023Reasons to Reject? Aligning Language Models with Judgmentscited by 15
  6. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  7. 2024Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLMcited by 14
  8. 2022Towards Boosting the Open-Domain Chatbot with Human Feedbackcited by 11
  9. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  10. 2023Improving Open Language Models by Learning from Organic Interactionscited by 10
  11. 2023Let Me Teach You: Pedagogical Foundations of Feedback for Language Modelscited by 9
  12. 2023Learning From Free-Text Human Feedback -- Collect New Datasets Or Extend Existing Ones?cited by 7
  13. 2024PersonalityChat: Conversation Distillation for Personalized Dialog Modeling with Facts and Traitscited by 6
  14. 2023System-Level Natural Language Feedbackcited by 4
  15. 2024Learning from Implicit User Feedback, Emotions and Demographic Information in Task-Oriented and Document-Grounded Dialoguescited by 3
  16. 2023What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perceptioncited by 2
  17. 2024Learning Goal-Conditioned Representations for Language Reward Modelscited by 2
  18. 2023Social Commonsense-Guided Search Query Generation for Open-Domain Knowledge-Powered Conversationscited by 1
  19. 2023Cognitively Inspired Components for Social Conversational Agentscited by 1
  20. 2026CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Productioncited by 1
  21. 2023Hexa: Self-Improving for Knowledge-Grounded Dialogue Systemcited by 0
  22. 2023Learning to love diligent trolls: Accounting for rater effects in the dialogue safety taskcited by 0
  23. 2025Towards Automated Error Discovery: A Study in Conversational AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Human-centric Dialog Training via Offline Reinforcement Learning
  2. 2025Playpen: An Environment for Exploring Learning Through Conversational Interaction
  3. 2026User-Aware Active Knowledge Acquisition for Emotional Support Dialogue
  4. 2024Naturally Occurring Feedback is Common, Extractable and Useful
  5. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Data
  6. 2026Verifiable Self-Evolution for Open-Ended Dialogue Skills via Future-Feedback Prediction
  7. 2024The Future of Open Human Feedback
  8. 2024Rewriting Conversational Utterances with Instructed Large Language Models