# PrefReward: Learning User Preference Matrix for Personalized Text Generation

arXiv: https://arxiv.org/abs/2607.21067 · HTML: https://mlsanity.com/paper/2607.21067/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 9 of its 13 references that are in the map.
- First posted on arXiv 23 Jul 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 9 of 9)

- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training](https://mlsanity.com/paper/2306.03166/) (2023)
- [Personalized Text Generation with Contrastive Activation Steering](https://mlsanity.com/paper/2503.05213/) (2025)
- [Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs](https://mlsanity.com/paper/2502.19148/) (2025)
- [Context Steering: Controllable Personalization at Inference Time](https://mlsanity.com/paper/2405.01768/) (2024)
- [Personalized Text Generation with Fine-Grained Linguistic Control](https://mlsanity.com/paper/2402.04914/) (2024)
- [Drift: Decoding-time Personalized Alignments with Implicit User Preferences](https://mlsanity.com/paper/2502.14289/) (2025)

## Built on by (most cited first, 1 of 1)

- [Personalize at Test Time: Learning User Preferences for Image Generation](https://mlsanity.com/paper/2610.09015/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable ability in generating personalized content by leveraging user histories and contextual cues. However, most existing personalization approaches rely on implicit representations within model parameters, making it difficult to interpret user-specific preferences or effectively handle long-context dependencies. To address these challenges, we propose PrefReward, a novel preference-aware generative framework that explicitly models user styles through a structured preference matrix and integrates it into the decoding process as a reward signal. PrefReward consists of two stages: (1) extracting a user-specific preference matrix that summarizes individual stylistic tendencies, and (2) using the matrix to guide generation via a KL-divergence-based reward function. Experiments on the LongLaMP dataset show that PrefReward outperforms non-personalized and retrieval-based baselines in both generation quality and personalization interpretability.