mlsanityData to 7 Oct 2026

Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment

arXiv 2504.12663 ↗First posted 17 Apr 2025cs.CL cs.AIcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Aligning language models with human preferences presents significant challenges, particularly in achieving personalization without incurring excessive computational costs. Existing methods rely on reward signals and additional annotated data, limiting their scalability and adaptability to diverse human values. To address these challenges, we introduce Persona-judge, a novel discriminative paradigm that enables training-free personalized alignment with unseen preferences. Instead of optimizing policy parameters through external reward feedback, Persona-judge leverages the intrinsic preference judgment capabilities of the model. Specifically, a draft model generates candidate tokens conditioned on a given preference, while a judge model, embodying another preference, cross-validates the predicted tokens whether to be accepted. Experimental results demonstrate that Persona-judge, using the inherent preference evaluation mechanisms of the model, offers a scalable and computationally efficient solution to personalized alignment, paving the way for more adaptive customized alignment. Our code is available here.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 61 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

47 of 61 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment19 Mar
  2. 2024PAD: Personalized Alignment of LLMs at Decoding-Time5 Oct
  3. 2024Aligning to Thousands of Preferences via System Message Generalization28 May
  4. 2024A Roadmap to Pluralistic Alignment7 Feb
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  6. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct
  7. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models23 Aug
  8. 2023Aligning Language Models to User Opinions24 May
  9. 2020Learning to summarize from human feedback2 Sep

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  12. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
26 more
  1. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  2. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  3. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  4. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  5. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  6. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  7. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  8. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  9. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  10. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
  11. 2024Can LLM be a Personalized Judge?cited by 99
  12. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  13. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  14. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
  15. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  16. 2023Fast Model Debias with Machine Unlearningcited by 70
  17. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  18. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 44
  19. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
  20. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  21. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
  22. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  23. 2024BiasAlert: A Plug-and-play Tool for Social Bias Detection in LLMscited by 21
  24. 2024MORL-Prompt: An Empirical Analysis of Multi-Objective Reinforcement Learning for Discrete Prompt Optimizationcited by 16
  25. 2024Large Language Model Bias Mitigation from the Perspective of Knowledge Editingcited by 16
  26. 2025Capturing Individual Human Preferences with Reward Featurescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2026Me-Agent: A Personalized Mobile Agent with Two-Level User Habit Learning for Enhanced Interactioncited by 6
  2. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Modelscited by 4
  3. 2025Towards Proactive Personalization through Profile Customization for Individual Users in Dialoguescited by 3
  4. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuningcited by 2
  5. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  6. 2025Detection, Classification, and Mitigation of Gender Bias in Large Language Modelscited by 1
  7. 2026Light Alignment Improves LLM Safety via Model Self-Reflection with a Single Neuroncited by 0
  8. 2026Why Expert Alignment Is Hard: Evidence from Subjective Evaluationcited by 0
  9. 2026Whose Norms? Disentangling Cultural and Personal Alignment in Large Language Modelscited by 0
  10. 2026Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensionscited by 0
  11. 2026From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
  2. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  3. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  4. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignment
  5. 2025Drift: Decoding-time Personalized Alignments with Implicit User Preferences
  6. 2025HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation
  7. 2025Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?
  8. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis