# Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models

arXiv: https://arxiv.org/abs/2609.35695 · HTML: https://mlsanity.com/paper/2609.35695/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 19 of 19)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024)
- [Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown](https://mlsanity.com/paper/2410.00847/) (2024)
- [Group Preference Optimization: Few-Shot Alignment of Large Language Models](https://mlsanity.com/paper/2310.11523/) (2023)
- [PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences](https://mlsanity.com/paper/2406.08469/) (2024)
- [GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment](https://mlsanity.com/paper/2410.08193/) (2024)
- [Cascade Reward Sampling for Efficient Decoding-Time Alignment](https://mlsanity.com/paper/2406.16306/) (2024)
- [LoRe: Personalizing LLMs via Low-Rank Reward Modeling](https://mlsanity.com/paper/2504.14439/) (2025)
- [Language Model Personalization via Reward Factorization](https://mlsanity.com/paper/2503.06358/) (2025)
- [Instant Personalized Large Language Model Adaptation via Hypernetwork](https://mlsanity.com/paper/2510.16282/) (2025)
- [The Era of Real-World Human Interaction: RL from User Conversations](https://mlsanity.com/paper/2509.25137/) (2025)
- [T-POP: Test-Time Personalization with Online Preference Feedback](https://mlsanity.com/paper/2509.24696/) (2025)
- [Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization](https://mlsanity.com/paper/2604.07343/) (2026)

## Abstract (arXiv)

Aligning large language models (LLMs) to diverse user preferences is fundamentally hindered by standard alignment paradigms that optimize for monolithic users. In this work, empirical studies are first used to reveal the existence of a massive, untapped performance headroom for personalized generation through test-time alignment. We demonstrate that personalized generation is uniquely suited for test-time scaling methods like Best-of-N (BoN) because it can be viewed primarily as a candidate matching problem rather than a generator capability bottleneck. While reward models could in principle exploit this headroom, they are poorly calibrated for personalization, and their billion-parameter scale makes scoring large candidate pools prohibitively expensive. To overcome this limitation, we propose a parameter-efficient framework utilizing million-parameter scale multi-layer perceptron (MLP) ranking models. Our personalized ranking model directly reuses the internal embeddings of the base generator with minimal overhead. By scaling train-time data to provide fine-grained personalized preferences, this million-parameter ranking model accurately scores large candidate pools and can seamlessly guide generation to reduce the cost of materializing N candidates. Extensive experiments on nine datasets spanning three personalized generation settings show that our personalized ranking model effectively exploits the discovered headroom, outperforming billion-parameter generalist reward models on every dataset, with under 0.4% of their parameters and four orders of magnitude lower scoring latency.