mlsanityData to 7 Oct 2026

Yo'LLaVA: Your Personalized Language and Vision Assistant

arXiv 2406.09400 ↗First posted 13 Jun 2024cs.CV cs.LGcited by 61

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multimodal Models (LMMs) have shown remarkable capabilities across a variety of tasks (e.g., image captioning, visual question answering). While broad, their knowledge remains generic (e.g., recognizing a dog), and they are unable to handle personalized subjects (e.g., recognizing a user's pet dog). Human reasoning, in contrast, typically operates within the context of specific subjects in our surroundings. For example, one might ask, "What should I buy for my dog's birthday?"; as opposed to a generic inquiry about "What should I buy for a dog's birthday?". Similarly, when looking at a friend's image, the interest lies in seeing their activities (e.g., "my friend is holding a cat"), rather than merely observing generic human actions (e.g., "a man is holding a cat"). In this paper, we introduce the novel task of personalizing LMMs, so that they can have conversations about a specific subject. We propose Yo'LLaVA, which learns to embed a personalized subject into a set of latent tokens given a handful of example images of the subject. Our qualitative and quantitative analyses reveal that Yo'LLaVA can learn the concept more efficiently using fewer tokens and more effectively encode the visual attributes compared to strong prompting baselines (e.g., LLaVA).

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 43 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

31 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2023Mistral 7Bcited by 3,065
  8. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2022Visual Prompt Tuningcited by 1,674
16 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  4. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  6. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  7. 2017Sampling Matters in Deep Embedding Learningcited by 285
  8. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  9. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  10. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  11. 2017Smart Mining for Deep Metric Learningcited by 96
  12. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  13. 2020You Impress Me: Dialogue Generation via Mutual Persona Perceptioncited by 85
  14. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  15. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  16. 2023Visual Instruction Inversion: Image Editing via Visual Promptingcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 28; 2026: 27.

61 papers.

Sort:

By area:

  1. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  2. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  3. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 53
  4. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  5. 2025MMKE-Bench: A Multimodal Editing Benchmark for Diverse Visual Knowledgecited by 22
  6. 2025YoChameleon: Personalized Vision and Language Generationcited by 20
  7. 2026M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactionscited by 20
  8. 2024Personalized Multimodal Large Language Models: A Surveycited by 19
  9. 2025Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sinkcited by 15
  10. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  11. 2025Personalized Vision via Visual In-Context Learningcited by 14
  12. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
38 more
  1. 2025RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Modelscited by 13
  2. 2024Personalized Large Vision-Language Modelscited by 12
  3. 2025Training-Free Personalization via Retrieval and Reasoning on Fingerprintscited by 10
  4. 2026PersonaVLM: Long-Term Personalized Multimodal LLMscited by 10
  5. 2024Personalized Representation from Personalized Generationcited by 9
  6. 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Modelscited by 8
  7. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  8. 2025Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakescited by 7
  9. 2025POET: Supporting Prompting Creativity and Personalization with Automated Expansion of Text-to-Image Generationcited by 6
  10. 2025Online-PVLM: Advancing Personalized VLMs with Online Concept Learningcited by 5
  11. 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
  12. 2025Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLMcited by 3
  13. 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrievalcited by 3
  14. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuningcited by 3
  15. 2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoningcited by 2
  16. 2025Aligning VLM Assistants with Personalized Situated Cognitioncited by 2
  17. 2025Improving Personalized Search with Regularized Low-Rank Parameter Updatescited by 2
  18. 2025Multimodal Policy Internalization for Conversational Agentscited by 2
  19. 2025Personalized Image Descriptions from Attention Sequencescited by 2
  20. 2026Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Userscited by 2
  21. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 2
  22. 2026Personal Visual Memory from Explicit and Implicit Evidencecited by 2
  23. 2025Region-Level Context-Aware Multimodal Understandingcited by 1
  24. 2025MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understandingcited by 1
  25. 2025Relational Visual Similaritycited by 1
  26. 2026IIR-VLM: In-Context Instance-level Recognition for Large Vision-Language Modelscited by 1
  27. 2026Contextualized Visual Personalization in Vision-Language Modelscited by 1
  28. 2026GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistantcited by 1
  29. 2025MC-LLaVA: Multi-Concept Personalized Vision-Language Modelcited by 0
  30. 2025Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgettingcited by 0
  31. 2025EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correctioncited by 0
  32. 2025Interpretability Framework for LLMs in Undergraduate Calculuscited by 0
  33. 2026PALM-Bench: A Comprehensive Benchmark for Personalized Audio-Language Modelscited by 0
  34. 2026Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actionscited by 0
  35. 2026Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognitioncited by 0
  36. 2026How to Take a Memorable Picture? Empowering Users with Actionable Feedbackcited by 0
  37. 2026Ego: Embedding-Guided Personalization of Vision-Language Modelscited by 0
  38. 2026Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contextscited by 0

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models
  2. 2024Matryoshka Multimodal Models
  3. 2024Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
  4. 2025Spatial-LLaVA: Enhancing Large Language Models with Spatial Referring Expressions for Visual Understanding
  5. 2025Guiding Multimodal Large Language Models with Blind and Low Vision People Visual Questions for Proactive Visual Interpretations
  6. 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
  7. 2025How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
  8. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey