mlsanityData to 7 Oct 2026

User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal

arXiv 2507.23158 ↗First posted 30 Jul 2025cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Once language models (LMs) are deployed, they can interact with users long-term, ideally evolving based on their feedback. Asking for direct user feedback can be disruptive; thus, we study harvesting implicit user feedback from user-LM interaction logs. We study two user-LM interaction datasets (WildChat and LMSYS). First, we analyze user feedback in the user-LLM conversation logs, providing insights into when and why such feedback occurs. Second, we study harvesting learning signals from such implicit user feedback. Specifically, we study whether incorporating the contents of user feedback (e.g., user wanted clarification), in addition to the polarity of the feedback, can improve the model performance. We observe mixed results, showing this helps in short human-designed questions (MTBench) but not on longer and more complex questions (WildBench). Together, we provide an in-depth study of implicit user feedback, showing its potential and limitations.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

22 of 30 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  5. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  7. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  9. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  10. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  11. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  12. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
10 more
  1. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  2. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  3. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 93
  4. 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 70
  5. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  6. 2025ChatBench: From Static Benchmarks to Human-AI Evaluationcited by 31
  7. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 27
  8. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  9. 2024Naturally Occurring Feedback is Common, Extractable and Usefulcited by 14
  10. 2023Let Me Teach You: Pedagogical Foundations of Feedback for Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 15.

19 papers.

Sort:

By area:

  1. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 10
  2. 2025PrefDisco: Benchmarking Proactive Personalized Reasoningcited by 7
  3. 2026CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbackscited by 6
  4. 2025DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learningcited by 5
  5. 2026Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learningcited by 5
  6. 2025Attention to Non-Adopterscited by 3
  7. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  8. 2026Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigmcited by 2
  9. 2026ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignmentcited by 2
  10. 2025Black Box Absorption: LLMs Undermining Innovative Ideascited by 1
  11. 2026Security and Privacy Prompts in the Wild: What Users Ask LLMs and How LLMs Respondcited by 1
  12. 2026From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Languagecited by 1
7 more
  1. 2026Learning from 53.6K Real-World Developer Edits of AI-Generated Codecited by 1
  2. 2026ReIn: Conversational Error Recovery with Reasoning Inceptioncited by 0
  3. 2026Demonstration of Pneuma-Seeker: Agentic System for Reifying and Fulfilling Information Needs on Tabular Datacited by 0
  4. 2026Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Modelscited by 0
  5. 2026Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluationcited by 0
  6. 2026Feedback-Assisted Trust Propagation over Document Relation Graphs for Retrieval-Augmented Generationcited by 0
  7. 2026User Feedback Provides a Unique Signal that LLMs Can not Detectcited by 0