mlsanityData to 7 Oct 2026

Ego: Embedding-Guided Personalization of Vision-Language Models

arXiv 2603.09771 ↗First posted 10 Mar 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

AI assistants that support humans in daily life are becoming increasingly feasible, driven by the rapid advancements in multimodal language models. A key challenge lies in overcoming the generic nature of these models to deliver personalized experiences. Existing approaches to personalizing large vision language models often rely on additional training stages, which limit generality and scalability, or on engineered pipelines with external pre-trained modules, which hinder deployment efficiency. In this work, we propose an efficient personalization method that leverages the model's inherent ability to capture personalized concepts. Specifically, we extract visual tokens that predominantly represent the target concept by utilizing the model's internal attention mechanisms. These tokens serve as a memory of that specific concept, enabling the model to recall and describe it when it appears in test images. We conduct a comprehensive and unified evaluation of our approach and SOTA methods across various personalization settings including single-concept, multi-concept, and video personalization, demonstrating strong performance gains with minimal personalization overhead.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 39 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

38 of 39 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Online In-Context Distillation for Low-Resource Vision Language Models20 Oct
  2. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models22 Sep
  3. 2025Personalization Toolkit: Training Free Personalization of Large Vision Language Models4 Feb
  4. 2024Personalized Large Vision-Language Models23 Dec
  5. 2024Personalized Visual Instruction Tuning9 Oct
  6. 2024MyVLM: Personalizing VLMs for User-Specific Queries21 Mar
  7. 2023Meta-Personalizing Vision-Language Models to Find Named Instances in Video16 Jun

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2025Qwen2.5-VL Technical Reportcited by 5,418
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2017Billion-scale similarity search with GPUscited by 2,477
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
19 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  5. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  6. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  7. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  8. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  9. 2023Large Language Models in Law: A Surveycited by 128
  10. 2024Personalization of Large Language Models: A Surveycited by 100
  11. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  12. 2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 61
  13. 2024What Makes Multimodal In-Context Learning Work?cited by 54
  14. 2023Exploring Large Language Model for Graph Data Understanding in Online Job Recommendationscited by 45
  15. 2024RAP: Retrieval-Augmented Personalization for Multimodal Large Language Modelscited by 24
  16. 2025True Multimodal In-Context Learning Needs Attention to the Visual Contextcited by 14
  17. 2025VisiPruner: Decoding Discontinuous Cross-Modal Dynamics for Efficient Multimodal LLMscited by 14
  18. 2025Training-Free Personalization via Retrieval and Reasoning on Fingerprintscited by 10
  19. 2025Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MMPB: It's Time for Multi-Modal Personalization
  2. 2026Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models
  3. 2025MM-R1: Unleashing the Power of Unified Multimodal Large Language Models for Personalized Image Generation
  4. 2026Contextualized Visual Personalization in Vision-Language Models
  5. 2024MC-LLaVA: Multi-Concept Personalized Vision-Language Model
  6. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuning
  7. 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrieval
  8. 2026Language-Instructed Vision Embeddings for Controllable and Generalizable Perception