mlsanityData to 7 Oct 2026

Personalized Visual Instruction Tuning

arXiv 2410.07113 ↗First posted 9 Oct 2024cs.CVcited by 24

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advancements in multimodal large language models (MLLMs) have demonstrated significant progress; however, these models exhibit a notable limitation, which we refer to as "face blindness". Specifically, they can engage in general conversations but fail to conduct personalized dialogues targeting at specific individuals. This deficiency hinders the application of MLLMs in personalized settings, such as tailored visual assistants on mobile devices, or domestic robots that need to recognize members of the family. In this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel data curation and training framework designed to enable MLLMs to identify target individuals within an image and engage in personalized and coherent dialogues. Our approach involves the development of a sophisticated pipeline that autonomously generates training data containing personalized conversations. This pipeline leverages the capabilities of various visual experts, image generation models, and (multi-modal) large language models. To evaluate the personalized potential of MLLMs, we present a benchmark called P-Bench, which encompasses various question types with different levels of difficulty. The experiments demonstrate a substantial personalized performance enhancement after fine-tuning with our curated dataset.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 43 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

36 of 43 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
24 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  3. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  4. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  5. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  7. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 431
  8. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  9. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  10. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  11. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  12. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  13. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  14. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
  15. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  16. 2023Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Modelscited by 85
  17. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  18. 2018Engaging Image Captioning Via Personalitycited by 51
  19. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  20. 2023HiLM-D: Enhancing MLLMs with Multi-Scale High-Resolution Details for Autonomous Drivingcited by 48
  21. 2024Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptionscited by 43
  22. 2017Attend to You: Personalized Image Captioning with Context Sequence Memory Networkscited by 35
  23. 2019Automatic Generation of Personalized Comment Based on User Profilecited by 9
  24. 2023User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 8.

24 papers.

Sort:

By area:

  1. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  2. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  3. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  4. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
  5. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  6. 2024Personalized Multimodal Large Language Models: A Surveycited by 19
  7. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
  8. 2025RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Modelscited by 13
  9. 2024Personalized Large Vision-Language Modelscited by 12
  10. 2025ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialectscited by 11
  11. 2025Training-Free Personalization via Retrieval and Reasoning on Fingerprintscited by 10
  12. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
12 more
  1. 2026PersonaVLM: Long-Term Personalized Multimodal LLMscited by 10
  2. 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Modelscited by 8
  3. 2024Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Codecited by 3
  4. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuningcited by 3
  5. 2025Personalized Image Descriptions from Attention Sequencescited by 2
  6. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 2
  7. 2025MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 0
  8. 2026Ego: Embedding-Guided Personalization of Vision-Language Modelscited by 0
  9. 2026Personalizing MLLMs via Reinforced Multimodal Reference Gamecited by 0
  10. 2026StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuningcited by 0
  11. 2026Group Preference Collapse in Personalized Multimodal Large Language Modelscited by 0
  12. 2026Video2Skill: From Streaming Experience to Reusable Embodied Skillscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023SVIT: Scaling up Visual Instruction Tuning
  2. 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
  3. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidance
  4. 2025MMPB: It's Time for Multi-Modal Personalization
  5. 2024SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant
  6. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
  7. 2025TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistant
  8. 2023Visual Instruction Tuning