# WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

arXiv: https://arxiv.org/abs/2408.15549 · HTML: https://mlsanity.com/paper/2408.15549/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 28 Aug 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 39)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint](https://mlsanity.com/paper/2312.11456/) (2023)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores](https://mlsanity.com/paper/2311.09766/) (2023)
- [Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models](https://mlsanity.com/paper/2407.07263/) (2024)

## Built on by (most cited first, 30 of 31)

- [Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale](https://mlsanity.com/paper/2504.14225/) (2025)
- [PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory](https://mlsanity.com/paper/2512.06688/) (2025)
- [HaluMem: Evaluating Hallucinations in Memory Systems of Agents](https://mlsanity.com/paper/2511.03506/) (2025)
- [Aligning Language Models from User Interactions](https://mlsanity.com/paper/2603.12273/) (2026)
- [Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment](https://mlsanity.com/paper/2505.15456/) (2025)
- [CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions](https://mlsanity.com/paper/2508.01674/) (2025)
- [Experiential Reinforcement Learning](https://mlsanity.com/paper/2602.13949/) (2026)
- [Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generation](https://mlsanity.com/paper/2503.16789/) (2025)
- [Aligning Large Language Models with Implicit Preferences from User-Generated Content](https://mlsanity.com/paper/2506.04463/) (2025)
- [When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents](https://mlsanity.com/paper/2507.09329/) (2025)
- [OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models](https://mlsanity.com/paper/2508.21061/) (2025)
- [ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions](https://mlsanity.com/paper/2605.20087/) (2026)
- [Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors](https://mlsanity.com/paper/2605.07847/) (2026)
- [DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learning](https://mlsanity.com/paper/2510.02341/) (2025)
- [GMTRouter: Personalized LLM Router over Multi-turn User Interactions](https://mlsanity.com/paper/2511.08590/) (2025)
- [Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents](https://mlsanity.com/paper/2602.01405/) (2026)
- [Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs](https://mlsanity.com/paper/2509.23166/) (2025)
- [WildReward: Learning Reward Models from In-the-Wild Human Interactions](https://mlsanity.com/paper/2602.08829/) (2026)
- [GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memory](https://mlsanity.com/paper/2607.05543/) (2026)
- [Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversations](https://mlsanity.com/paper/2503.08035/) (2025)
- [A Framework for Situating Innovations, Opportunities, and Challenges in Advancing Vertical Systems with Large AI Models](https://mlsanity.com/paper/2504.02793/) (2025)
- [DeltaMem: Towards Agentic Memory Management via Reinforcement Learning](https://mlsanity.com/paper/2604.01560/) (2026)
- [Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans](https://mlsanity.com/paper/2512.23693/) (2025)
- [Improve Large Language Model Systems with User Logs](https://mlsanity.com/paper/2602.06470/) (2026)
- [Adopt neq Adapt: Longitudinal Analyses of LLM Conversations in the Wild](https://mlsanity.com/paper/2605.29018/) (2026)
- [Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policies](https://mlsanity.com/paper/2511.05018/) (2025)
- [Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users](https://mlsanity.com/paper/2606.20482/) (2026)
- [Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models](https://mlsanity.com/paper/2608.09109/) (2026)
- [User Feedback Provides a Unique Signal that LLMs Can not Detect](https://mlsanity.com/paper/2609.02859/) (2026)
- [HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization](https://mlsanity.com/paper/2609.09835/) (2026)

## Abstract (arXiv)

As large language models (LLMs) continue to advance, aligning these models with human preferences has emerged as a critical challenge. Traditional alignment methods, relying on human or LLM annotated datasets, are limited by their resource-intensive nature, inherent subjectivity, misalignment with real-world user preferences, and the risk of feedback loops that amplify model biases. To overcome these limitations, we introduce WildFeedback, a novel framework that leverages in-situ user feedback during conversations with LLMs to create preference datasets automatically. Given a corpus of multi-turn user-LLM conversation, WildFeedback identifies and classifies user feedback to LLM responses between conversation turns. The user feedback is then used to create examples of preferred and dispreferred responses according to users' preference. Our experiments demonstrate that LLMs fine-tuned on WildFeedback dataset exhibit significantly improved alignment with user preferences, as evidenced by both traditional benchmarks and our proposed checklist-guided evaluation. By incorporating in-situ feedback from actual users, WildFeedback addresses the scalability, subjectivity, and bias challenges that plague existing approaches, marking a significant step toward developing LLMs that are more responsive to the diverse and evolving needs of their users.