mlsanityData to 7 Oct 2026

Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment

arXiv 2412.20834 ↗First posted 30 Dec 2024cs.CL cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Aligning Large Language Models (LLMs) with general human preferences has been proved crucial in improving the interaction quality between LLMs and human. However, human values are inherently diverse among different individuals, making it insufficient to align LLMs solely with general preferences. To address this, personalizing LLMs according to individual feedback emerges as a promising solution. Nonetheless, this approach presents challenges in terms of the efficiency of alignment algorithms. In this work, we introduce a flexible paradigm for individual preference alignment. Our method fundamentally improves efficiency by disentangling preference representation from text generation in LLMs. We validate our approach across multiple text generation tasks and demonstrate that it can produce aligned quality as well as or better than PEFT-based methods, while reducing additional training time for each new individual preference by 80% to 90% in comparison with them.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 64 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

43 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Aligning to Thousands of Preferences via System Message Generalization28 May
  2. 2024Personalized Language Modeling from Personalized Human Feedback6 Feb
  3. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging17 Oct
  4. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Models12 Oct
  5. 2023Human Feedback is not Gold Standard28 Sep
  6. 2020Learning to summarize from human feedback2 Sep

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
25 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  4. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  6. 2017Toward Controlled Generation of Textcited by 271
  7. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  8. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  9. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  10. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  11. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  12. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  13. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  14. 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
  15. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  16. 2021Transformer-based Conditional Variational Autoencoder for Controllable Story Generationcited by 44
  17. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  18. 2022DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generationcited by 35
  19. 2024Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statementscited by 33
  20. 2019Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoderscited by 27
  21. 2024Aligning Crowd Feedback via Distributional Preference Reward Modelingcited by 26
  22. 2022Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generationcited by 20
  23. 2020MixPoet: Diverse Poetry Generation via Learning Controllable Mixed Latent Spacecited by 17
  24. 2021Finetuning Pretrained Transformers into Variational Autoencoderscited by 14
  25. 2022Improving Variational Autoencoders with Density Gap-based Regularizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026AI and My Values: User Perceptions of LLMs' Ability to Extract, Embody, and Explain Human Values from Casual Conversationscited by 2
  2. 2026Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Userscited by 2
  3. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
  4. 2026Structuring the Space of Sociotechnical Alignmentcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  2. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
  3. 2024Aligning LLMs with Individual Preferences via Interaction
  4. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  5. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
  6. 2024A Survey on Human Preference Learning for Large Language Models
  7. 2025Aligning Large Language Models with Implicit Preferences from User-Generated Content
  8. 2025Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation