# Personalized Multimodal Large Language Models: A Survey

arXiv: https://arxiv.org/abs/2412.02142 · HTML: https://mlsanity.com/paper/2412.02142/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 87 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 32)

- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance](https://mlsanity.com/paper/2406.07209/) (2024)
- [LongLaMP: A Benchmark for Personalized Long-form Text Generation](https://mlsanity.com/paper/2407.11016/) (2024)
- [Yo'LLaVA: Your Personalized Language and Vision Assistant](https://mlsanity.com/paper/2406.09400/) (2024)
- [MyVLM: Personalizing VLMs for User-Specific Queries](https://mlsanity.com/paper/2403.14599/) (2024)
- [InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignment](https://mlsanity.com/paper/2402.08785/) (2024)
- [λ-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space](https://mlsanity.com/paper/2402.05195/) (2024)
- [A Content-Driven Micro-Video Recommendation Dataset at Scale](https://mlsanity.com/paper/2309.15379/) (2023)
- [MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation](https://mlsanity.com/paper/2404.05674/) (2024)
- [Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models](https://mlsanity.com/paper/2404.04243/) (2024)
- [Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2408.09698/) (2024)
- [Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond](https://mlsanity.com/paper/2402.10805/) (2024)
- [Personalized Visual Instruction Tuning](https://mlsanity.com/paper/2410.07113/) (2024)
- [MMREC: LLM Based Multi-Modal Recommender System](https://mlsanity.com/paper/2408.04211/) (2024)
- [Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models](https://mlsanity.com/paper/2402.08670/) (2024)
- [EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot](https://mlsanity.com/paper/2406.15177/) (2024)
- [UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion](https://mlsanity.com/paper/2401.13388/) (2024)
- [Exploring Recommendation Capabilities of GPT-4V(ision): A Preliminary Case Study](https://mlsanity.com/paper/2311.04199/) (2023)
- [MM-Diff: High-Fidelity Image Personalization via Multi-Modal Condition Integration](https://mlsanity.com/paper/2403.15059/) (2024)
- [Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design](https://mlsanity.com/paper/2407.08882/) (2024)
- [CoMMIT: Coordinated Multimodal Instruction Tuning](https://mlsanity.com/paper/2407.20454/) (2024)
- [RA-Rec: An Efficient ID Representation Alignment Framework for LLM-based Recommendation](https://mlsanity.com/paper/2402.04527/) (2024)
- [AlignBot: Aligning VLM-powered Customized Task Planning with User Reminders Through Fine-Tuning for Household Robots](https://mlsanity.com/paper/2409.11905/) (2024)
- [End-to-end training of Multimodal Model and ranking Model](https://mlsanity.com/paper/2404.06078/) (2024)
- [X-Reflect: Cross-Reflection Prompting for Multimodal Recommendation](https://mlsanity.com/paper/2408.15172/) (2024)
- [Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm](https://mlsanity.com/paper/2408.08693/) (2024)
- [A Learnable Agent Collaboration Network Framework for Personalized Multimodal AI Search Engine](https://mlsanity.com/paper/2409.00636/) (2024)
- [A Multimodal In-Context Tuning Approach for E-Commerce Product Description Generation](https://mlsanity.com/paper/2402.13587/) (2024)
- [Language-Guided Self-Supervised Video Summarization Using Text Semantic Matching Considering the Diversity of the Video](https://mlsanity.com/paper/2405.08890/) (2024)

## Built on by (most cited first, 19 of 19)

- [A Survey of Personalized Large Language Models: Progress and Future Directions](https://mlsanity.com/paper/2502.11528/) (2025)
- [Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment](https://mlsanity.com/paper/2505.15456/) (2025)
- [A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations](https://mlsanity.com/paper/2505.14106/) (2025)
- [Skill-R1: Agent Skill Evolution via Reinforcement Learning](https://mlsanity.com/paper/2605.09359/) (2026)
- [PersonaVLM: Long-Term Personalized Multimodal LLMs](https://mlsanity.com/paper/2604.13074/) (2026)
- [JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model](https://mlsanity.com/paper/2504.03770/) (2025)
- [WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning](https://mlsanity.com/paper/2509.04744/) (2025)
- [ClimateLLM: Efficient Weather Forecasting via Frequency-Aware Large Language Models](https://mlsanity.com/paper/2502.11059/) (2025)
- [PEARL: Personalized Streaming Video Understanding Model](https://mlsanity.com/paper/2603.20422/) (2026)
- [Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.02913/) (2026)
- [HUMORCHAIN: Theory-Guided Multi-Stage Reasoning for Interpretable Multimodal Humor Generation](https://mlsanity.com/paper/2511.21732/) (2025)
- [AMPS: Adaptive Modality Preference Steering via Functional Entropy](https://mlsanity.com/paper/2602.12533/) (2026)
- [Aligning VLM Assistants with Personalized Situated Cognition](https://mlsanity.com/paper/2506.00930/) (2025)
- [PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behavior](https://mlsanity.com/paper/2507.18447/) (2025)
- [OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents](https://mlsanity.com/paper/2605.11169/) (2026)
- [Personal Visual Memory from Explicit and Implicit Evidence](https://mlsanity.com/paper/2605.28806/) (2026)
- [ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant](https://mlsanity.com/paper/2505.03654/) (2025)
- [Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck](https://mlsanity.com/paper/2605.08526/) (2026)
- [F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking](https://mlsanity.com/paper/2605.12995/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have become increasingly important due to their state-of-the-art performance and ability to integrate multiple data modalities, such as text, images, and audio, to perform complex tasks with high accuracy. This paper presents a comprehensive survey on personalized multimodal large language models, focusing on their architecture, training methods, and applications. We propose an intuitive taxonomy for categorizing the techniques used to personalize MLLMs to individual users, and discuss the techniques accordingly. Furthermore, we discuss how such techniques can be combined or adapted when appropriate, highlighting their advantages and underlying rationale. We also provide a succinct summary of personalization tasks investigated in existing research, along with the evaluation metrics commonly used. Additionally, we summarize the datasets that are useful for benchmarking personalized MLLMs. Finally, we outline critical open challenges. This survey aims to serve as a valuable resource for researchers and practitioners seeking to understand and advance the development of personalized multimodal large language models.