mlsanityData to 7 Oct 2026

MC-LLaVA: Multi-Concept Personalized Vision-Language Model

arXiv 2411.11706 ↗First posted 18 Nov 2024cs.CV cs.AIcited by 53

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Current vision-language models (VLMs) show exceptional abilities across diverse tasks, such as visual question answering. To enhance user experience, recent studies have investigated VLM personalization to understand user-provided concepts. However, they mainly focus on single concepts, neglecting the existence and interplay of multiple concepts, which limits real-world applicability. This paper proposes MC-LLaVA, a multi-concept personalization paradigm. Specifically, MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step. To reduce the training costs, we propose a personalized textual prompt that uses visual token information to initialize concept tokens. Additionally, we introduce a personalized visual prompt during inference, aggregating location maps for enhanced recognition and grounding capabilities. To further push the performance upper bound, we incorporate an optional auxiliary loss, better enhancing the proposed personalized prompts. To decorate the VLM personalization research, we contribute a high-quality dataset. We carefully collect images with multiple characters and objects from movies and manually create question-answer samples for multi-concept scenarios, featuring superior diversity. Comprehensive experiments demonstrate that MC-LLaVA achieves impressive multi-concept personalized responses, paving the way for VLMs to become better user assistants. The code and dataset will be released at hrefhttps://github.com/arctanxarc/MC-LLaVAhttps://github.com/arctanxarc/MC-LLaVA.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 54 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

41 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Personalized Visual Instruction Tuning9 Oct
  2. 2024Are Bigger Encoders Always Better in Vision Large Models?1 Aug
  3. 2024MyVLM: Personalizing VLMs for User-Specific Queries21 Mar

Other references · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
26 more
  1. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  2. 2022Visual Prompt Tuningcited by 1,674
  3. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  6. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  7. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  8. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  9. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  10. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  11. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  12. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  13. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  14. 2022From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Modelscited by 114
  15. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  16. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  17. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  18. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  19. 2024Large Language Models Empowered Personalized Web Agentscited by 64
  20. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 61
  21. 2023Knowledge-Augmented Large Language Models for Personalized Contextual Query Suggestioncited by 43
  22. 2024LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Modelcited by 43
  23. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  24. 2024Prospect Personalized Recommendation on Large Language Model-based Agent Platformcited by 19
  25. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  26. 2023MoSA: Mixture of Sparse Adapters for Visual Efficient Tuningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 32; 2026: 18.

53 papers.

Sort:

By area:

  1. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  2. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 89
  3. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 69
  4. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 66
  5. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  6. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  7. 2025Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMscited by 41
  8. 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learningcited by 32
  9. 2024OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generationcited by 24
  10. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  11. 2025Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Maskingcited by 21
  12. 2026M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactionscited by 20
38 more
  1. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 19
  2. 2025SCALAR: Scale-wise Controllable Visual Autoregressive Learningcited by 18
  3. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14
  4. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  5. 2025Rethinking Driving World Model as Synthetic Data Generator for Perception Taskscited by 14
  6. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  7. 2025CrossLMM: Decoupling Long Video Sequences from LMMs via Dual Cross-Attention Mechanismscited by 11
  8. 2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
  9. 2025IAG: Input-aware Backdoor Attack on VLM-based Visual Groundingcited by 11
  10. 2025Text2SQL-Flow: A Robust SQL-Aware Data Augmentation Framework for Text-to-SQLcited by 11
  11. 2025Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modelingcited by 10
  12. 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
  13. 2025MICo-150K: A Comprehensive Dataset Advancing Multi-Image Compositioncited by 9
  14. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  15. 2026Unveiling the Cognitive Compass: Theory-of-Mind-Guided Multimodal Emotion Reasoningcited by 8
  16. 2026PEARL: Personalized Streaming Video Understanding Modelcited by 8
  17. 2025Faithful-First Reasoning, Planning, and Acting for Multimodal LLMscited by 7
  18. 2025Semantic Context Matters: Improving Conditioning for Autoregressive Modelscited by 7
  19. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
  20. 2026Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspectivecited by 6
  21. 2025Eevee: Towards Close-up High-resolution Video-based Virtual Try-oncited by 5
  22. 2026AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAGcited by 5
  23. 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
  24. 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrievalcited by 3
  25. 2026Research on World Models Is Not Merely Injecting World Knowledge into Specific Taskscited by 3
  26. 2026SlowBA: An efficiency backdoor attack towards VLM-based GUI agentscited by 3
  27. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuningcited by 3
  28. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  29. 2025VCU-Bridge: Hierarchical Visual Connotation Understanding via Semantic Bridgingcited by 2
  30. 2025GRAN-TED: Generating Robust, Aligned, and Nuanced Text Embedding for Diffusion Modelscited by 2
  31. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 2
  32. 2024Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Modelcited by 1
  33. 2025Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Explorationcited by 1
  34. 2026IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Modelscited by 1
  35. 2026TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videoscited by 1
  36. 2026Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognitioncited by 0
  37. 2026Towards Customized Multimodal Role-Playcited by 0
  38. 2026Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Modelscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MC-LLaVA: Multi-Concept Personalized Vision-Language Model
  2. 2024SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant
  3. 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Models
  4. 2024Teaching VLMs to Localize Specific Objects from In-context Examples
  5. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  6. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering
  7. 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
  8. 2024Visual Question Answering Instruction: Unlocking Multimodal Large Language Model To Domain-Specific Visual Multitasks