# User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal

arXiv: https://arxiv.org/abs/2507.23158 · HTML: https://mlsanity.com/paper/2507.23158/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint](https://mlsanity.com/paper/2312.11456/) (2023)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [HelpSteer2: Open-source dataset for training top-performing reward models](https://mlsanity.com/paper/2406.08673/) (2024)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [Recursive Introspection: Teaching Language Model Agents How to Self-Improve](https://mlsanity.com/paper/2407.18219/) (2024)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Learning from Dialogue after Deployment: Feed Yourself, Chatbot!](https://mlsanity.com/paper/1901.05415/) (2019)
- [Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs](https://mlsanity.com/paper/2311.09469/) (2023)
- [Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questions](https://mlsanity.com/paper/2410.13788/) (2024)
- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [ChatBench: From Static Benchmarks to Human-AI Evaluation](https://mlsanity.com/paper/2504.07114/) (2025)
- [Navigating Rifts in Human-LLM Grounding: Study and Benchmark](https://mlsanity.com/paper/2503.13975/) (2025)
- [Leveraging Implicit Feedback from Deployment Data in Dialogue](https://mlsanity.com/paper/2307.14117/) (2023)
- [Naturally Occurring Feedback is Common, Extractable and Useful](https://mlsanity.com/paper/2407.10944/) (2024)
- [Let Me Teach You: Pedagogical Foundations of Feedback for Language Models](https://mlsanity.com/paper/2307.00279/) (2023)

## Built on by (most cited first, 19 of 19)

- [Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directions](https://mlsanity.com/paper/2602.22680/) (2026)
- [PrefDisco: Benchmarking Proactive Personalized Reasoning](https://mlsanity.com/paper/2510.00177/) (2025)
- [CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks](https://mlsanity.com/paper/2603.18736/) (2026)
- [DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning](https://mlsanity.com/paper/2510.02341/) (2025)
- [Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning](https://mlsanity.com/paper/2603.04597/) (2026)
- [Attention to Non-Adopters](https://mlsanity.com/paper/2510.15951/) (2025)
- [WildReward: Learning Reward Models from In-the-Wild Human Interactions](https://mlsanity.com/paper/2602.08829/) (2026)
- [ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment](https://mlsanity.com/paper/2603.23184/) (2026)
- [Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigm](https://mlsanity.com/paper/2602.11661/) (2026)
- [Black Box Absorption: LLMs Undermining Innovative Ideas](https://mlsanity.com/paper/2510.20612/) (2025)
- [ReIn: Conversational Error Recovery with Reasoning Inception](https://mlsanity.com/paper/2602.17022/) (2026)
- [Security and Privacy Prompts in the Wild: What Users Ask LLMs and How LLMs Respond](https://mlsanity.com/paper/2606.18062/) (2026)
- [From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Language](https://mlsanity.com/paper/2607.16232/) (2026)
- [Learning from 53.6K Real-World Developer Edits of AI-Generated Code](https://mlsanity.com/paper/2607.25130/) (2026)
- [Demonstration of Pneuma-Seeker: Agentic System for Reifying and Fulfilling Information Needs on Tabular Data](https://mlsanity.com/paper/2604.14422/) (2026)
- [Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models](https://mlsanity.com/paper/2608.09109/) (2026)
- [Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation](https://mlsanity.com/paper/2608.29887/) (2026)
- [Feedback-Assisted Trust Propagation over Document Relation Graphs for Retrieval-Augmented Generation](https://mlsanity.com/paper/2609.00543/) (2026)
- [User Feedback Provides a Unique Signal that LLMs Can not Detect](https://mlsanity.com/paper/2609.02859/) (2026)

## Abstract (arXiv)

Once language models (LMs) are deployed, they can interact with users long-term, ideally evolving based on their feedback. Asking for direct user feedback can be disruptive; thus, we study harvesting implicit user feedback from user-LM interaction logs. We study two user-LM interaction datasets (WildChat and LMSYS). First, we analyze user feedback in the user-LLM conversation logs, providing insights into when and why such feedback occurs. Second, we study harvesting learning signals from such implicit user feedback. Specifically, we study whether incorporating the contents of user feedback (e.g., user wanted clarification), in addition to the polarity of the feedback, can improve the model performance. We observe mixed results, showing this helps in short human-designed questions (MTBench) but not on longer and more complex questions (WildBench). Together, we provide an in-depth study of implicit user feedback, showing its potential and limitations.