# One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment

arXiv: https://arxiv.org/abs/2601.18731 · HTML: https://mlsanity.com/paper/2601.18731/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 5 of its 8 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 5 of 5)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [MeLU: Meta-Learned User Preference Estimator for Cold-Start Recommendation](https://mlsanity.com/paper/1908.00413/) (2019)
- [Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision](https://mlsanity.com/paper/2402.02658/) (2024)
- [Meta-Learning with Adaptive Weighted Loss for Imbalanced Cold-Start Recommendation](https://mlsanity.com/paper/2302.14640/) (2023)

## Built on by (most cited first, 5 of 5)

- [Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization](https://mlsanity.com/paper/2604.00997/) (2026)
- [Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling](https://mlsanity.com/paper/2609.35109/) (2026)
- [Learning to Simulate Individuals from Macro Social Signals](https://mlsanity.com/paper/2610.07062/) (2026)
- [Personalize at Test Time: Learning User Preferences for Image Generation](https://mlsanity.com/paper/2610.09015/) (2026)
- [Online LLM Personalization from Pairwise Preferences Across Users](https://mlsanity.com/paper/2610.17203/) (2026)

## Abstract (arXiv)

Alignment of Large Language Models (LLMs) aims to align outputs with human preferences, and personalized alignment further adapts models to individual users. This relies on personalized reward models that capture user-specific preferences and automatically provide individualized feedback. However, developing these models faces two critical challenges: the scarcity of feedback from individual users and the need for efficient adaptation to unseen users. We argue that addressing these constraints requires a paradigm shift from fitting data to learn user preferences to learn the process of preference adaptation. To realize this, we propose Meta Reward Modeling (MRM), which reformulates personalized reward modeling as a meta-learning problem. Specifically, we represent each user's reward model as a weighted combination of base reward functions, and optimize the initialization of these weights using a Model-Agnostic Meta-Learning (MAML)-style framework to support fast adaptation under limited feedback. To ensure robustness, we introduce the Robust Personalization Objective (RPO), which places greater emphasis on hard-to-learn users during meta optimization. Extensive experiments on personalized preference datasets validate that MRM enhances few-shot personalization, improves user robustness, and consistently outperforms baselines. We release code at https://github.com/ModalityDance/MRM.