mlsanityData to 7 Oct 2026

MyVLM: Personalizing VLMs for User-Specific Queries

arXiv 2403.14599 ↗First posted 21 Mar 2024cs.CVcited by 58

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent large-scale vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and generating textual descriptions for visual content. However, these models lack an understanding of user-specific concepts. In this work, we take a first step toward the personalization of VLMs, enabling them to learn and reason over user-provided concepts. For example, we explore whether these models can learn to recognize you in an image and communicate what you are doing, tailoring the model to reflect your personal experiences and relationships. To effectively recognize a variety of user-specific concepts, we augment the VLM with external concept heads that function as toggles for the model, enabling the VLM to identify the presence of specific target concepts in a given image. Having recognized the concept, we learn a new concept embedding in the intermediate feature space of the VLM. This embedding is tasked with guiding the language model to naturally integrate the target concept in its generated response. We apply our technique to BLIP-2 and LLaVA for personalized image captioning and further show its applicability for personalized visual question-answering. Our experiments demonstrate our ability to generalize to unseen images of learned concepts while preserving the model behavior on unrelated inputs.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 76 of its 98 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

76 of 98 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  3. 2023Meta-Personalizing Vision-Language Models to Find Named Instances in Video16 Jun
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 72

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  13. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  14. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  15. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  16. 2016HyperNetworkscited by 1,003
  17. 2022Mass-Editing Memory in a Transformercited by 976
  18. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  19. 2023A Survey on Multimodal Large Language Modelscited by 898
  20. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  21. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  22. 2021Editing Factual Knowledge in Language Modelscited by 548
  23. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  24. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  25. 2023Erasing Concepts from Diffusion Modelscited by 530
  26. 2021Fast Model Editing at Scalecited by 491
  27. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  28. 2022Memory-Based Model Editing at Scalecited by 443
  29. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  30. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 431
  31. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  32. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  33. 2023Data Filtering Networkscited by 247
  34. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  35. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  36. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  37. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  38. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 24; 2026: 26.

58 papers.

Sort:

By area:

  1. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  2. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  3. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 61
  4. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  5. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
  6. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  7. 2025MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledgecited by 22
  8. 2025YoChameleon: Personalized Vision and Language Generationcited by 20
  9. 2026M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactionscited by 20
  10. 2024Personalized Multimodal Large Language Models: A Surveycited by 19
  11. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  12. 2025Personalized Vision via Visual In-Context Learningcited by 14
38 more
  1. 2024InstantRestore: Single-Step Personalized Face Restoration with Shared-Image Attentioncited by 13
  2. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
  3. 2025RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Modelscited by 13
  4. 2024Personalized Large Vision-Language Modelscited by 12
  5. 2025Training-Free Personalization via Retrieval and Reasoning on Fingerprintscited by 10
  6. 2026PersonaVLM: Long-Term Personalized Multimodal LLMscited by 10
  7. 2024Personalized Representation from Personalized Generationcited by 9
  8. 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Modelscited by 8
  9. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  10. 2026PEARL: Personalized Streaming Video Understanding Modelcited by 8
  11. 2025CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answeringcited by 6
  12. 2025USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactionscited by 6
  13. 2026Ego-Grounding for Personalized Question-Answering in Egocentric Videoscited by 6
  14. 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
  15. 2025Reasoning LLMs for User-Aware Multimodal Conversational Agentscited by 4
  16. 2025PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agentscited by 4
  17. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuningcited by 3
  18. 2025Improving Personalized Search with Regularized Low-Rank Parameter Updatescited by 2
  19. 2025ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraintscited by 2
  20. 2025Multimodal Policy Internalization for Conversational Agentscited by 2
  21. 2025Personalized Image Descriptions from Attention Sequencescited by 2
  22. 2025Directional Textual Inversion for Personalized Text-to-Image Generationcited by 2
  23. 2026Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Userscited by 2
  24. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 2
  25. 2026Personal Visual Memory from Explicit and Implicit Evidencecited by 2
  26. 2025Natural Language Generation from Visual Events: State-of-the-Art and Key Open Questionscited by 1
  27. 2025Region-Level Context-Aware Multimodal Understandingcited by 1
  28. 2026IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Modelscited by 1
  29. 2026Contextualized Visual Personalization in Vision-Language Modelscited by 1
  30. 2025MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 0
  31. 2025Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgettingcited by 0
  32. 2026PALM-Bench: A Comprehensive Benchmark for Personalized Audio-Language Modelscited by 0
  33. 2026Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognitioncited by 0
  34. 2026Ego: Embedding-Guided Personalization of Vision-Language Modelscited by 0
  35. 2026Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrievalcited by 0
  36. 2026Towards Customized Multimodal Role-Playcited by 0
  37. 2026Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Modelscited by 0
  38. 2026Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalizationcited by 0

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Online-PVLM: Advancing Personalized VLMs with Online Concept Learning
  2. 2024Teaching VLMs to Localize Specific Objects from In-context Examples
  3. 2024Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluation
  4. 2025Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection
  5. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  6. 2025Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLM
  7. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  8. 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrieval