mlsanityData to 7 Oct 2026

Personalized Multimodal Large Language Models: A Survey

arXiv 2412.02142 ↗First posted 3 Dec 2024cs.CV cs.AI cs.CLcited by 19

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have become increasingly important due to their state-of-the-art performance and ability to integrate multiple data modalities, such as text, images, and audio, to perform complex tasks with high accuracy. This paper presents a comprehensive survey on personalized multimodal large language models, focusing on their architecture, training methods, and applications. We propose an intuitive taxonomy for categorizing the techniques used to personalize MLLMs to individual users, and discuss the techniques accordingly. Furthermore, we discuss how such techniques can be combined or adapted when appropriate, highlighting their advantages and underlying rationale. We also provide a succinct summary of personalization tasks investigated in existing research, along with the evaluation metrics commonly used. Additionally, we summarize the datasets that are useful for benchmarking personalized MLLMs. Finally, we outline critical open challenges. This survey aims to serve as a valuable resource for researchers and practitioners seeking to understand and advance the development of personalized multimodal large language models.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 87 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

32 of 87 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024CoMMIT: Coordinated Multimodal Instruction Tuning29 Jul
  2. 2024Yo'LLaVA: Your Personalized Language and Vision Assistant13 Jun
  3. 2024MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation8 Apr
  4. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep

Other references · 28

  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  2. 2024MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidancecited by 142
  3. 2024LongLaMP: A Benchmark for Personalized Long-form Text Generationcited by 91
  4. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  5. 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
  6. 2024λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Spacecited by 51
  7. 2023A Content-Driven Micro-Video Recommendation Dataset at Scalecited by 43
  8. 2024Identity Decoupling for Multi-Subject Personalization of Text-to-Image Modelscited by 33
  9. 2024Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendationcited by 33
  10. 2024Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyondcited by 27
  11. 2024Personalized Visual Instruction Tuningcited by 24
  12. 2024MMREC: LLM Based Multi-Modal Recommender Systemcited by 22
16 more
  1. 2024Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Modelscited by 18
  2. 2024EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbotcited by 17
  3. 2024UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusioncited by 15
  4. 2023Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Studycited by 12
  5. 2024MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integrationcited by 10
  6. 2024Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Designcited by 10
  7. 2024RA-Rec: An Efficient ID Representation Alignment Framework for LLM-based Recommendationcited by 7
  8. 2024AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robotscited by 7
  9. 2024End-to-end training of Multimodal Model and ranking Modelcited by 6
  10. 2024X-Reflect: Cross-Reflection Prompting for Multimodal Recommendationcited by 6
  11. 2024Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigmcited by 5
  12. 2024A Learnable Agent Collaboration Network Framework for Personalized Multimodal AI Search Enginecited by 5
  13. 2024A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generationcited by 4
  14. 2024Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Videocited by 3
  15. 2024User-Friendly Customized Generation with Multi-Modal Promptscited by 1
  16. 2024Layout-and-Retouch: A Dual-stage Framework for Improving Diversity in Personalized Image Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  2. 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignmentcited by 22
  3. 2025A Personalized Conversational Benchmark: Towards Simulating Personalized Conversationscited by 16
  4. 2026Skill-R1: Agent Skill Evolution via Reinforcement Learningcited by 12
  5. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
  6. 2025WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoningcited by 10
  7. 2026PersonaVLM: Long-Term Personalized Multimodal LLMscited by 10
  8. 2025ClimateLLM: Efficient Weather Forecasting via Frequency-Aware Large Language Modelscited by 9
  9. 2026PEARL: Personalized Streaming Video Understanding Modelcited by 8
  10. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  11. 2025HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generationcited by 4
  12. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
7 more
  1. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  2. 2025PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behaviorcited by 2
  3. 2026OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agentscited by 2
  4. 2026Personal Visual Memory from Explicit and Implicit Evidencecited by 2
  5. 2025ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistantcited by 1
  6. 2026Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneckcited by 1
  7. 2026F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Rankingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  2. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  3. 2024The Revolution of Multimodal Large Language Models: A Survey
  4. 2025Self-Improvement in Multimodal Large Language Models: A Survey
  5. 2024Efficient Multimodal Large Language Models: A Survey
  6. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  7. 2023Multimodal Large Language Models: A Survey
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models