mlsanityData to 14 Oct 2026

The Era of Real-World Human Interaction: RL from User Conversations

arXiv 2509.25137 ↗First posted 29 Sep 2025cs.AI cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

We posit that to achieve continual model improvement and multifaceted alignment, future models must learn from natural human interaction. Current conversational models are aligned using pre-annotated, expert-generated human feedback. In this work, we introduce Reinforcement Learning from Human Interaction (RLHI), a paradigm that learns directly from in-the-wild user conversations. We develop two complementary methods: (1) RLHI with User-Guided Rewrites, which revises unsatisfactory model outputs based on users' natural-language follow-up responses, (2) RLHI with User-Based Rewards, which learns via a reward model conditioned on knowledge of the user's long-term interaction history (termed persona). Together, these methods link long-term user personas to turn-level preferences via persona-conditioned preference optimization. Trained on conversations derived from WildChat, both RLHI variants outperform strong baselines in personalization and instruction-following, and similar feedback enhances performance on reasoning benchmarks. These results suggest organic human interaction offers scalable, effective supervision for personalized alignment.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 38 of its 43 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

38 of 43 references.

References · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
26 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  2. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  3. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  4. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  5. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
  6. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  7. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  8. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  9. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  10. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
  11. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 123
  12. 2024Personalization of Large Language Models: A Surveycited by 104
  13. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  14. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentationcited by 99
  15. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  16. 2023Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrieverscited by 92
  17. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  18. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  19. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 74
  20. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 60
  21. 2024Following Length Constraints in Instructionscited by 46
  22. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  23. 2024Naturally Occurring Feedback is Common, Extractable and Usefulcited by 14
  24. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13
  25. 2023Improving Open Language Models by Learning from Organic Interactionscited by 10
  26. 2024Retrospective Learning from Interactionscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  2. 2026ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactionscited by 7
  3. 2026Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agentscited by 4
  4. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  5. 2026CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Productioncited by 1
  6. 2026From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Languagecited by 1
  7. 2026Learning from 53.6K Real-World Developer Edits of AI-Generated Codecited by 1
  8. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  9. 2026MindZero: Learning Online Mental Reasoning With Zero Annotationscited by 0
  10. 2026Expectation Alignment of Language Models for Real-World User Expectationscited by 0
  11. 2026User Feedback Provides a Unique Signal that LLMs Can not Detectcited by 0
  12. 2026HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalizationcited by 0
1 more
  1. 2026Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward
  2. 2026Aligning Language Models from User Interactions
  3. 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  5. 2025Language Model Personalization via Reward Factorization
  6. 2025REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation
  7. 2026Learning User Simulators with Turing Rewards
  8. 2024Self-Evolved Reward Learning for LLMs