mlsanityData to 7 Oct 2026

WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback

arXiv 2408.15549 ↗First posted 28 Aug 2024cs.CLcited by 31

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

As large language models (LLMs) continue to advance, aligning these models with human preferences has emerged as a critical challenge. Traditional alignment methods, relying on human or LLM annotated datasets, are limited by their resource-intensive nature, inherent subjectivity, misalignment with real-world user preferences, and the risk of feedback loops that amplify model biases. To overcome these limitations, we introduce WildFeedback, a novel framework that leverages in-situ user feedback during conversations with LLMs to create preference datasets automatically. Given a corpus of multi-turn user-LLM conversation, WildFeedback identifies and classifies user feedback to LLM responses between conversation turns. The user feedback is then used to create examples of preferred and dispreferred responses according to users' preference. Our experiments demonstrate that LLMs fine-tuned on WildFeedback dataset exhibit significantly improved alignment with user preferences, as evidenced by both traditional benchmarks and our proposed checklist-guided evaluation. By incorporating in-situ feedback from actual users, WildFeedback addresses the scalability, subjectivity, and bias challenges that plague existing approaches, marking a significant step toward developing LLMs that are more responsive to the diverse and evolving needs of their users.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 54 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

39 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec
  2. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedback2 Oct
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2022When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels28 Oct

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
23 more
  1. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  2. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  4. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  5. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  6. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  8. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  9. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  10. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  11. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  12. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  13. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  14. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  15. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 69
  16. 2022POTATO: The Portable Text Annotation Toolcited by 64
  17. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
  18. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferencescited by 53
  19. 2023Safer-Instruct: Aligning Language Models with Automated Preference Datacited by 27
  20. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Levelcited by 22
  21. 2024Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Modelscited by 19
  22. 2024Blind Spots and Biases: Exploring the Role of Annotator Cognitive Biases in NLPcited by 11
  23. 2023S3-DST: Structured Open-Domain Dialogue Segmentation and State Tracking in the Era of LLMscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 15.

31 papers.

Sort:

By area:

  1. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 163
  2. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
  3. 2025HaluMem: Evaluating Hallucinations in Memory Systems of Agentscited by 51
  4. 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignmentcited by 22
  5. 2026Experiential Reinforcement Learningcited by 21
  6. 2026Aligning Language Models from User Interactionscited by 21
  7. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 20
  8. 2025Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generationcited by 10
  9. 2025Aligning Large Language Models with Implicit Preferences from User-Generated Contentcited by 9
  10. 2025When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agentscited by 7
  11. 2025OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Modelscited by 7
  12. 2025DRIFT: Learning from Abundant User Dissatisfaction in Real-World Preference Learningcited by 5
19 more
  1. 2025GMTRouter: Personalized LLM Router over Multi-turn User Interactionscited by 5
  2. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  3. 2026Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviorscited by 4
  4. 2026ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactionscited by 4
  5. 2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
  6. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  7. 2026GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memorycited by 3
  8. 2025Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversationscited by 2
  9. 2025A Framework for Situating Innovations, Opportunities, and Challenges in Advancing Vertical Systems with Large AI Modelscited by 2
  10. 2026DeltaMem: Towards Agentic Memory Management via Reinforcement Learningcited by 2
  11. 2025Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spanscited by 1
  12. 2026Improve Large Language Model Systems with User Logscited by 1
  13. 2026Adopt neq Adapt: Longitudinal Analyses of LLM Conversations in the Wildcited by 1
  14. 2025Pluralistic Behavior Suite: Stress-Testing Multi-Turn Adherence to Custom Behavioral Policiescited by 0
  15. 2026Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Userscited by 0
  16. 2026Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Modelscited by 0
  17. 2026User Feedback Provides a Unique Signal that LLMs Can not Detectcited by 0
  18. 2026HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalizationcited by 0
  19. 2026"Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverablescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning LLMs with Individual Preferences via Interaction
  2. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
  3. 2024Aligning to Thousands of Preferences via System Message Generalization
  4. 2024Dissecting Human and LLM Preferences
  5. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation
  6. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  7. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  8. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences