mlsanityData to 7 Oct 2026

Naturally Occurring Feedback is Common, Extractable and Useful

arXiv 2407.10944 ↗First posted 15 Jul 2024cs.CLcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Human feedback data is a critical component in developing language models. However, collecting this feedback is costly and ultimately not scalable. Inspired by the way human interlocutors provide spontaneous unsolicited feedback to each other, we propose to extract feedback that users naturally include when interacting with chat models. We manually annotated conversations to confirm the presence of naturally occurring feedback in a standard corpus, finding that as much as 30% of the chats include explicit feedback. Comparing to older datasets, we find that naturally occurring feedback is more prevalent in recent conversation datasets, suggesting that more than ever, naturally occurring feedback can serve as a valuable resource for feedback data. We propose a method for automatically extracting this feedback, and apply it to over 1M conversations to obtain hundreds of thousands of feedback samples. The extracted feedback shows promise: training with it improves over baseline models and enhances model alignment to human preferences.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 56 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

29 of 56 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LLM Evaluators Recognize and Favor Their Own Generations15 Apr
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference7 Mar
  3. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels28 Oct

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
14 more
  1. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  5. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  6. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  7. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  8. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  9. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  10. 2024LAB: Large-Scale Alignment for ChatBotscited by 39
  11. 2024Genie: Achieving Human Parity in Content-Grounded Datasets Generationcited by 31
  12. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Languagecited by 22
  13. 2023Learning to Simulate Natural Language Feedback for Interactive Semantic Parsingcited by 19
  14. 2024Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Inputcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wildcited by 24
  2. 2026Aligning Language Models from User Interactionscited by 21
  3. 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
  4. 2025The Era of Real-World Human Interaction: RL from User Conversationscited by 13
  5. 2024FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedbackcited by 11
  6. 2024The Future of Open Human Feedbackcited by 10
  7. 2026Mediocrity is the key for LLM as a Judge Anchor Selectioncited by 6
  8. 2025DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learningcited by 5
  9. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  10. 2025Show or Tell? Modeling the evolution of request-making in Human-LLM conversationscited by 3
  11. 2026CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Productioncited by 1
  12. 2026From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Languagecited by 1
2 more
  1. 2026User Feedback Provides a Unique Signal that LLMs Can not Detectcited by 0
  2. 2026AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
  2. 2023Learning From Free-Text Human Feedback -- Collect New Datasets Or Extend Existing Ones?
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback
  4. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
  5. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
  6. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Data
  7. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
  8. 2022Training Language Models with Language Feedback