mlsanityData to 14 Oct 2026

YoChameleon: Personalized Vision and Language Generation

arXiv 2504.20998 ↗First posted 29 Apr 2025cs.CV cs.AIcited by 20

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into powerful tools with millions of users. However, they remain generic models and lack personalized knowledge of specific user concepts. Previous work has explored personalization for text generation, yet it remains unclear how these methods can be adapted to new modalities, such as image generation. In this paper, we introduce Yo'Chameleon, the first attempt to study personalization for large multimodal models. Given 3-5 images of a particular concept, Yo'Chameleon leverages soft-prompt tuning to embed subject-specific information to (i) answer questions about the subject and (ii) recreate pixel-level details to produce images of the subject in new contexts. Yo'Chameleon is trained with (i) a self-prompting optimization mechanism to balance performance across multiple modalities, and (ii) a ``soft-positive" image generation approach to enhance image quality in a few-shot setting.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 64 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

55 of 64 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performance21 Oct
  2. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation8 Jul
  3. 2024Yo'LLaVA: Your Personalized Language and Vision Assistant13 Jun
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Segment Anythingcited by 8,234
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
36 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  8. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  9. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  10. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
  11. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 781
  12. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  13. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  14. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  15. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  16. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  17. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 416
  18. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 345
  19. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  20. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 336
  21. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  22. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 240
  23. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 212
  24. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  25. 2020Improving Image Captioning with Better Use of Captionscited by 84
  26. 2020Hard negative examples are hard, but usefulcited by 82
  27. 2023Inserting Anybody in Diffusion Models via Celeb Basiscited by 63
  28. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  29. 2020PANDORA Talks: Personality and Demographics on Redditcited by 40
  30. 2024Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidancecited by 34
  31. 2024Imagine yourself: Tuning-Free Personalized Image Generationcited by 30
  32. 2023RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generationcited by 25
  33. 2023Controllable Textual Inversion for Personalized Text-to-Image Generationcited by 14
  34. 2023Generate Anything Anywhere in Any Scenecited by 14
  35. 2021Learning Implicit User Profiles for Personalized Retrieval-Based Chatbotcited by 12
  36. 2016Bootstrapping Face Detection with Hard Negative Examplescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 11.

20 papers.

Sort:

By area:

  1. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  2. 2026M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactionscited by 25
  3. 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generationcited by 9
  4. 2026PEARL: Personalized Streaming Video Understanding Modelcited by 9
  5. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  6. 2026CoCo: Code as CoT for Text-to-Image Preview and Rare Concept Generationcited by 6
  7. 2025Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLMcited by 4
  8. 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrievalcited by 4
  9. 2025Multimodal Policy Internalization for Conversational Agentscited by 2
  10. 2025Personalized Image Descriptions from Attention Sequencescited by 2
  11. 2026Contextualized Visual Personalization in Vision-Language Modelscited by 2
  12. 2026Personal Visual Memory from Explicit and Implicit Evidencecited by 2
8 more
  1. 2025DCoAR: Deep Concept Injection into Unified Autoregressive Models for Personalized Text-to-Image Generationcited by 1
  2. 2025Relational Visual Similaritycited by 1
  3. 2026Obliviate: Erasing Concepts from Autoregressive Image Generation Modelscited by 1
  4. 2025YNTP-100: A Benchmark for Your Next Token Prediction with 100 Peoplecited by 0
  5. 2026Towards Customized Multimodal Role-Playcited by 0
  6. 2026Personal AI Agent for Camera Roll VQAcited by 0
  7. 2026Personalizing MLLMs via Reinforced Multimodal Reference Gamecited by 0
  8. 2026Group Preference Collapse in Personalized Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation
  2. 2022On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization
  3. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Models
  4. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Model
  5. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
  6. 2025PrefGen: Multimodal Preference Learning for Preference-Conditioned Image Generation
  7. 2025Personalized Text-to-Image Generation with Auto-Regressive Models
  8. 2025Personalized Reward Modeling for Text-to-Image Generation