mlsanityData to 7 Oct 2026

RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models

arXiv 2410.13360 ↗First posted 17 Oct 2024cs.CV cs.AI cs.CLcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

The development of large language models (LLMs) has significantly enhanced the capabilities of multimodal LLMs (MLLMs) as general assistants. However, lack of user-specific knowledge still restricts their application in human's daily life. In this paper, we introduce the Retrieval Augmented Personalization (RAP) framework for MLLMs' personalization. Starting from a general MLLM, we turn it into a personalized assistant in three steps. (a) Remember: We design a key-value database to store user-related information, e.g., user's name, avatar and other attributes. (b) Retrieve: When the user initiates a conversation, RAP will retrieve relevant information from the database using a multimodal retriever. (c) Generate: The input query and retrieved concepts' information are fed into MLLMs to generate personalized, knowledge-augmented responses. Unlike previous methods, RAP allows real-time concept editing via updating the external database. To further improve generation quality and alignment with user-specific information, we design a pipeline for data collection and create a specialized dataset for personalized training of MLLMs. Based on the dataset, we train a series of MLLMs as personalized multimodal assistants. By pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual concepts without additional finetuning. Our models demonstrate outstanding flexibility and generation quality across a variety of tasks, such as personalized image captioning, question answering and visual recognition. The code, data and models are available at https://hoar012.github.io/RAP-Project/.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 63 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

52 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  2. 2024Yo'LLaVA: Your Personalized Language and Vision Assistant13 Jun
  3. 2023OneLLM: One Framework to Align All Modalities with Language6 Dec
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
35 more
  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2017Billion-scale similarity search with GPUscited by 2,477
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  8. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  9. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  11. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  12. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  13. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  14. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  15. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  16. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  17. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 282
  18. 2023Inpaint Anything: Segment Anything Meets Image Inpaintingcited by 233
  19. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  20. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  21. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  22. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  23. 2020TAO: A Large-Scale Benchmark for Tracking Any Objectcited by 146
  24. 2023Retrieval meets Long Context Large Language Modelscited by 114
  25. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  26. 2023SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusioncited by 72
  27. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  28. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  29. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  30. 2024Personalized Large Language Modelscited by 44
  31. 2023EVCap: Retrieval-Augmented Image Captioning with External Visual-Name Memory for Open-World Comprehensioncited by 32
  32. 2023Retrieval-augmented Image Captioningcited by 28
  33. 2023Meta-Personalizing Vision-Language Models to Find Named Instances in Videocited by 16
  34. 2024ERAGent: Enhancing Retrieval-Augmented Language Models with Improved Accuracy, Efficiency, and Personalizationcited by 15
  35. 2024Personalized Residuals for Concept-Driven Text-to-Image Generationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 16.

24 papers.

Sort:

By area:

  1. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  2. 2026PersonaVLM: Long-Term Personalized Multimodal LLMscited by 10
  3. 2026Unified Personalized Understanding, Generating and Editingcited by 8
  4. 2026PEARL: Personalized Streaming Video Understanding Modelcited by 8
  5. 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
  6. 2025Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrievalcited by 3
  7. 2026Personalize Your Large Vision-language Models With In-context Prompt Tuningcited by 3
  8. 2025FuncPoison: Poisoning Function Library to Hijack Multi-agent Autonomous Driving Systemscited by 2
  9. 2025Personalized Image Descriptions from Attention Sequencescited by 2
  10. 2026Personal Visual Memory from Explicit and Implicit Evidencecited by 2
  11. 2025ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistantcited by 1
  12. 2025Region-Level Context-Aware Multimodal Understandingcited by 1
12 more
  1. 2026Contextualized Visual Personalization in Vision-Language Modelscited by 1
  2. 2025Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgettingcited by 0
  3. 2026Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognitioncited by 0
  4. 2026Ego: Embedding-Guided Personalization of Vision-Language Modelscited by 0
  5. 2026Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Modelscited by 0
  6. 2026Uni-Synergy: Bridging Understanding and Generation for Personalized Reasoning via Co-operative Reinforcement Learningcited by 0
  7. 2026Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactionscited by 0
  8. 2026LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logscited by 0
  9. 2026Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Modelscited by 0
  10. 2026Parametric Multimodal User Memory: Storing What Captions Cannot Carrycited by 0
  11. 2026AutoConcept: Training-Free Concept-Guided Reranking for Metadata-Available Composed Image Retrievalcited by 0
  12. 2026Whose record is this? Diagnosing and authorizing record use in personalized multimodal modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Personalized Multimodal Large Language Models: A Survey
  2. 2023Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Models
  3. 2025BESPOKE: Benchmark for Search-Augmented Large Language Model Personalization via Diagnostic Feedback
  4. 2025Personalizing Large Language Models using Retrieval Augmented Generation and Knowledge Graph
  5. 2024On the Way to LLM Personalization: Learning to Remember User Conversations
  6. 2025Personalize Your LLM: Fake it then Align it
  7. 2025TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistant
  8. 2024Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation