# Reinforcement Learning from User Feedback

arXiv: https://arxiv.org/abs/2505.14946 · HTML: https://mlsanity.com/paper/2505.14946/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 8 of its 12 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 8 of 8)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [The Perfect Blend: Redefining RLHF with Mixture of Judges](https://mlsanity.com/paper/2409.20370/) (2024)
- [Monolith: Real Time Recommendation System With Collisionless Embedding Table](https://mlsanity.com/paper/2209.07663/) (2022)
- [Improving Open Language Models by Learning from Organic Interactions](https://mlsanity.com/paper/2306.04707/) (2023)

## Built on by (most cited first, 8 of 8)

- [Scaling Up Efficient Small Language Models Serving and Deployment for Semantic Job Search](https://mlsanity.com/paper/2510.22101/) (2025)
- [G-UBS: Towards Robust Understanding of Implicit Feedback via Group-Aware User Behavior Simulation](https://mlsanity.com/paper/2508.05709/) (2025)
- [Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigm](https://mlsanity.com/paper/2602.11661/) (2026)
- [Black Box Absorption: LLMs Undermining Innovative Ideas](https://mlsanity.com/paper/2510.20612/) (2025)
- [Improve Large Language Model Systems with User Logs](https://mlsanity.com/paper/2602.06470/) (2026)
- [Don't Let Bandit Feedback Pull Continual LLM-Recommender Updates Off Target](https://mlsanity.com/paper/2605.18899/) (2026)
- [Towards Better Agents for Multi-Turn User Interaction: The Next User Turn Is More Than Context](https://mlsanity.com/paper/2608.17499/) (2026)
- [Echo: Learning from Experience Data via User-Driven Refinement](https://mlsanity.com/paper/2605.21984/) (2026)

## Abstract (arXiv)

As large language models (LLMs) are increasingly deployed in diverse user facing applications, aligning them with real user preferences becomes essential. Existing methods like Reinforcement Learning from Human Feedback (RLHF) rely on expert annotators trained on manually defined guidelines, whose judgments may not reflect the priorities of everyday users. We introduce Reinforcement Learning from User Feedback (RLUF), a framework for aligning LLMs directly to implicit signals from users in production. RLUF addresses key challenges of user feedback: user feedback is often binary (e.g., emoji reactions), sparse, and occasionally adversarial. We train a reward model, P[Love], to predict the likelihood that an LLM response will receive a Love Reaction, a lightweight form of positive user feedback, and integrate P[Love] into a multi-objective policy optimization framework alongside helpfulness and safety objectives. In large-scale experiments, we show that P[Love] is predictive of increased positive feedback and serves as a reliable offline evaluator of future user behavior. Policy optimization using P[Love] significantly raises observed positive-feedback rates, including a 28% increase in Love Reactions during live A/B tests. However, optimizing for positive reactions introduces reward hacking challenges, requiring careful balancing of objectives. By directly leveraging implicit signals from users, RLUF offers a path to aligning LLMs with real-world user preferences at scale.