Group Preference Collapse in Personalized Multimodal Large Language Models
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Personalized multimodal large language models (MLLMs) aim to generate user-specific responses, but existing methods mainly rely on profile-level information and overlook diverse user preferences. We identify group preference collapse, where multi-user personalized MLLMs become insensitive to individual preferences and drift toward dominant population-level choices due to suppressed preference signals and unreliable preference use during generation. We propose PrefMoE, a preference-centric framework that separates stable profile information from preference-related representations. PrefMoE decomposes preferences into shared prototypes and personalized residuals, preserves individualized residuals with imbalance-aware learning, counterfactual pseudo-user augmentation, and residual decorrelation, and routes profile and preference factors through separate LoRA adaptation paths. Experiments across multiple MLLM backbones show that PrefMoE improves preference-sensitive personalization while substantially reducing preference collapse. Project page: https://prefmoe.github.io/.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 55 references that are in the map. First posted on arXiv 12 Jun 2026.
Builds on
42 of 55 references.
Closest ancestors · 6
Same direction, newest first
- 2025YoChameleon: Personalized Vision and Language Generation29 Apr
- 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
- 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
- 2024Yo'LLaVA: Your Personalized Language and Vision Assistant13 Jun
- 2023Improved Baselines with Visual Instruction Tuning5 Oct
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
Other references · 36
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2017Deep reinforcement learning from human preferencescited by 3,912
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2015VQA: Visual Question Answeringcited by 2,506
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
- 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
24 more
- 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
- 2016A Persona-Based Neural Conversation Modelcited by 223
- 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
- 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
- 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
- 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 54
- 2024Axioms for AI Alignment from Human Feedbackcited by 52
- 2025Boosting MLLM Reasoning with Text-Debiased Hint-GRPOcited by 32
- 2026M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactionscited by 25
- 2024Personalized Visual Instruction Tuningcited by 24
- 2024LOVA3: Learning to Visual Question Answering, Asking and Assessmentcited by 21
- 2025Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understandingcited by 20
- 2023Meta-Personalizing Vision-Language Models to Find Named Instances in Videocited by 16
- 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
- 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
- 2026PEARL: Personalized Streaming Video Understanding Modelcited by 9
- 2024TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddingscited by 8
- 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Modelscited by 8
- 2017Speaker Role Contextual Modeling for Language Understanding and Dialogue Policy Learningcited by 6
- 2025Online-PVLM: Advancing Personalized VLMs with Online Concept Learningcited by 5
- 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
- 2025Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLMcited by 4
- 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrievalcited by 4
- 2025MM-Prompt: Cross-Modal Prompt Tuning for Continual Visual Question Answeringcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
- 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
- 2026Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization
- 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
- 2024Personalized Language Modeling from Personalized Human Feedback
- 2025CoPL: Collaborative Preference Learning for Personalizing LLMs
- 2026Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework
- 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models