mlsanityData to 7 Oct 2026

Personalized Large Vision-Language Models

arXiv 2412.17610 ↗First posted 23 Dec 2024cs.CVcited by 12

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The personalization model has gained significant attention in image generation yet remains underexplored for large vision-language models (LVLMs). Beyond generic ones, with personalization, LVLMs handle interactive dialogues using referential concepts (e.g., ``Mike and Susan are talking.'') instead of the generic form (e.g., ``a boy and a girl are talking.''), making the conversation more customizable and referentially friendly. In addition, PLVM is equipped to continuously add new concepts during a dialogue without incurring additional costs, which significantly enhances the practicality. PLVM proposes Aligner, a pre-trained visual encoder to align referential concepts with the queried images. During the dialogues, it extracts features of reference images with these corresponding concepts and recognizes them in the queried image, enabling personalization. We note that the computational cost and parameter count of the Aligner are negligible within the entire framework. With comprehensive qualitative and quantitative analyses, we reveal the effectiveness and superiority of PLVM.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 48 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

42 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Personalized Visual Instruction Tuning9 Oct
  2. 2024MyVLM: Personalizing VLMs for User-Specific Queries21 Mar
  3. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
26 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  11. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  12. 2024InstantID: Zero-shot Identity-Preserving Generation in Secondscited by 431
  13. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  14. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  15. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  16. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  17. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  18. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  19. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  20. 2023Cones: Concept Neurons in Diffusion Models for Customized Generationcited by 150
  21. 2023VideoBooth: Diffusion-based Video Generation with Image Promptscited by 124
  22. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 61
  23. 2024Artemis: Towards Referential Understanding in Complex Videoscited by 25
  24. 2022Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migrationcited by 24
  25. 2024ChatterBox: Multi-round Multimodal Referring and Groundingcited by 24
  26. 2022HiViT: Hierarchical Vision Transformer Meets Masked Image Modelingcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  2. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
  3. 2025Training-Free Personalization via Retrieval and Reasoning on Fingerprintscited by 10
  4. 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Modelscited by 8
  5. 2025PersONAL: Towards a Comprehensive Benchmark for Personalized Embodied Agentscited by 4
  6. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuningcited by 3
  7. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  8. 2026Personal Visual Context Learning in Large Multimodal Modelscited by 2
  9. 2025ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistantcited by 1
  10. 2026Ego: Embedding-Guided Personalization of Vision-Language Modelscited by 0
  11. 2026Personalizing MLLMs via Reinforced Multimodal Reference Gamecited by 0
  12. 2026Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
  2. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback
  3. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Model
  4. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks
  5. 2025Improved Alignment of Modalities in Large Vision Language Models
  6. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs
  7. 2025ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models
  8. 2025Feedback-Driven Vision-Language Alignment with Minimal Human Supervision