mlsanityData to 13 Oct 2026

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

arXiv 2604.12479 ↗First posted 14 Apr 2026cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent advances in large language models (LLMs) have significantly improved the alignment of models with general human preferences. However, a major challenge remains in adapting LLMs to individual preferences, which are not only diverse but also dynamic. In this paper, we introduce a novel framework, Preference-Paired Fine-Tuning (PFT), designed to align models with contradictory and evolving individual preferences. We present a new dataset, Value Conflict Dilemma (VCD), which includes scenarios that involve conflicting human preferences, facilitating the evaluation of our approach. Our experiments demonstrate that PFT outperforms single-preference training methods, achieving up to 96.6% accuracy in multi-choice classification tasks and the highest open-ended generation score of 8.69. PFT also shows significant improvements over DPO, SFT and some traditional training methods, especially when handling conflicting preferences. Additionally, with limited user history data, models can inferring preference vector rapidly, achieving a 44.76% improvement in user-specific preference alignment in comparison to single-preference models.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 55 references that are in the map. First posted on arXiv 14 Apr 2026.

Builds on

35 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment25 Feb
  2. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences12 Jun
  3. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  4. 2024Hummer: Towards Limited Competitive Preference Dataset19 May
  5. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  6. 2024ARGS: Alignment as Reward-Guided Search23 Jan
  7. 2023Preference Ranking Optimization for Human Alignment30 Jun
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  3. 2017Deep reinforcement learning from human preferencescited by 3,871
  4. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,042
  5. 2018Behavioral Cloning from Observationcited by 483
  6. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  7. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  8. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  9. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  10. 2024Large Language Models as Urban Residents: An LLM Agent Framework for Personal Mobility Generationcited by 99
  11. 2024Can LLM be a Personalized Judge?cited by 99
  12. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
14 more
  1. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  2. 2024Large Language Models Empowered Personalized Web Agentscited by 64
  3. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 58
  4. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  5. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 45
  6. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 43
  7. 2025Controlling Large Language Models Through Concept Activation Vectorscited by 28
  8. 2024Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilitiescited by 22
  9. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedbackcited by 20
  10. 2025DecisionFlow: Advancing Large Language Model as Principled Decision Makercited by 12
  11. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  12. 2025CLASH: Evaluating Language Models on Judging High-Stakes Dilemmas from Multiple Perspectivescited by 10
  13. 2024FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guaranteescited by 6
  14. 2025Preference Learning for AI Alignment: a Causal Perspectivecited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Momentum-Coupled Rubric Adaptation for Detailed Image Captioningcited by 2
  2. 2026The Free-Recipe Limit: Every Recipe Effect Measures Which Premise of an Idealised Learner Brokecited by 0
  3. 2026Harness-Search: Guiding Long-Horizon Search through Multi-Agent Coordinationcited by 0
  4. 2026Learning to Simulate Individuals from Macro Social Signalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2024A Survey on Human Preference Learning for Large Language Models
  3. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  4. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
  5. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
  6. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  7. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignment
  8. 2025Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors