mlsanityData to 14 Oct 2026

PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning

arXiv 2508.14076 ↗First posted 12 Aug 2025cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

also in 1 more
Abstract (arXiv)

Reward models (RMs), which are central to existing post-training methods, aim to align LLM outputs with human values by providing feedback signals during fine-tuning. However, existing RMs struggle to capture nuanced, user-specific preferences, especially under limited data and across diverse domains. Thus, we introduce PersRM-R1, the first reasoning-based reward modeling framework specifically designed to identify and represent personal factors from only one or a few personal exemplars. To address challenges including limited data availability and the requirement for robust generalization, our approach combines synthetic data generation with a two-stage training pipeline consisting of supervised fine-tuning followed by reinforcement fine-tuning. Experimental results demonstrate that PersRM-R1 outperforms existing models of similar size and matches the performance of much larger models in both accuracy and generalizability, paving the way for more effective personalized LLMs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 44 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

34 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization28 May
  2. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  3. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  4. 2020Learning to summarize from human feedback2 Sep

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
18 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  5. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  6. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 377
  7. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 339
  8. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 262
  9. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  10. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  11. 2024Style Vectors for Steering Generative Large Language Modelcited by 56
  12. 2025Reward Reasoning Modelcited by 47
  13. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 43
  14. 2023Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logiccited by 35
  15. 2025Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedbackcited by 32
  16. 2023Internally Rewarded Reinforcement Learningcited by 14
  17. 2024PersLLM: A Personified Training Approach for Large Language Modelscited by 6
  18. 2024Beyond Introspection: Reinforcing Thinking via Externalist Behavioral Feedbackcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Predicting LLM Output Length via Entropy-Guided Representationscited by 9
  2. 2026Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2026Using Context Is Not Enough: Test-Time Training for Personalized Reward Modeling
  2. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  3. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoning
  4. 2024Semi-Supervised Reward Modeling via Iterative Self-Training
  5. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHF
  6. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modeling
  7. 2025Language Model Personalization via Reward Factorization
  8. 2026P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling