mlsanityData to 14 Oct 2026

MC-LLaVA: Multi-Concept Personalized Vision-Language Model

arXiv 2503.18854 ↗First posted 24 Mar 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Current vision-language models (VLMs) show exceptional abilities across diverse tasks, such as visual question answering. To enhance user experience, recent studies investigate VLM personalization to understand user-provided concepts. However, they mainly focus on single-concept personalization, neglecting the existence and interplay of multiple concepts, which limits real-world applicability. This paper proposes the first multi-concept personalization paradigm, MC-LLaVA. Specifically, MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step. To reduce the costs related to joint training, we propose a personalized textual prompt that uses visual token information to initialize concept tokens. Additionally, we introduce a personalized visual prompt during inference, aggregating location confidence maps for enhanced recognition and grounding capabilities. To advance multi-concept personalization research, we further contribute a high-quality instruction tuning dataset. We carefully collect images with multiple characters and objects from movies and manually generate question-answer samples for multi-concept scenarios, featuring superior diversity. Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses, paving the way for VLMs to become better user-specific assistants. The code and dataset will be publicly available at https://github.com/arctanxarc/MC-LLaVA}.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 54 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

43 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Models4 Feb
  2. 2024Personalized Visual Instruction Tuning9 Oct
  3. 2024Are Bigger Encoders Always Better in Vision Large Models?1 Aug
  4. 2024MyVLM: Personalizing VLMs for User-Specific Queries21 Mar

Other references · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
27 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  3. 2022Visual Prompt Tuningcited by 1,681
  4. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  8. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  10. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  11. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  12. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  13. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  14. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  15. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 115
  16. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  17. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 90
  18. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  19. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  20. 2024Large Language Models Empowered Personalized Web Agentscited by 66
  21. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 62
  22. 2024LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Modelcited by 44
  23. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
  24. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  25. 2024Prospect Personalized Recommendation on Large Language Model-based Agent Platformcited by 19
  26. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  27. 2023MoSA: Mixture of Sparse Adapters for Visual Efficient Tuningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Model
  2. 2024SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant
  3. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuning
  4. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step
  5. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering
  6. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  8. 2024Teaching VLMs to Localize Specific Objects from In-context Examples