mlsanityData to 7 Oct 2026

"This is my unicorn, Fluffy": Personalizing frozen vision-language representations

arXiv 2204.01694 ↗First posted 4 Apr 2022cs.CV cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

Large Vision & Language models pretrained on web-scale data provide representations that are invaluable for numerous V&L problems. However, it is unclear how they can be used for reasoning about user-specific visual concepts in unstructured language. This problem arises in multiple domains, from personalized image retrieval to personalized interaction with smart devices. We introduce a new learning setup called Personalized Vision & Language (PerVL) with two new benchmark datasets for retrieving and segmenting user-specific "personalized" concepts "in the wild". In PerVL, one should learn personalized concepts (1) independently of the downstream task (2) allowing a pretrained model to reason about them with free language, and (3) does not require personalized negative examples. We propose an architecture for solving PerVL that operates by extending the input vocabulary of a pretrained model with new word embeddings for the new personalized concepts. The model can then reason about them by simply using them in a sentence. We demonstrate that our approach learns personalized visual concepts from a few examples and can effectively apply them in image retrieval and semantic segmentation using rich textual queries.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 73 of its 93 references that are in the map. First posted on arXiv 4 Apr 2022.

Builds on

73 of 93 references.

References · 73

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2016Matching Networks for One Shot Learningcited by 2,246
  11. 2017Deep Setscited by 1,394
  12. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
38 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  2. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  3. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  4. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  8. 2022Language-driven Semantic Segmentationcited by 711
  9. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  10. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  12. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  13. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  14. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  15. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  16. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  17. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  18. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  19. 2021Pivotal Tuning for Latent-based Editing of Real Imagescited by 427
  20. 2020InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANscited by 393
  21. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  22. 2020StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flowscited by 348
  23. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  24. 2020Few-Shot Class-Incremental Learningcited by 249
  25. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 228
  26. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  27. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  28. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  29. 2020Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learningcited by 179
  30. 2015Label-Embedding for Image Classificationcited by 164
  31. 2021Reviving Iterative Training with Mask Guidance for Interactive Segmentationcited by 162
  32. 2019DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Imagescited by 132
  33. 2021Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learningcited by 114
  34. 2018Neural Baby Talkcited by 107
  35. 2021HyperInverter: Improving StyleGAN Inversion via Hypernetworkcited by 85
  36. 2021Learning to Compose Visual Relationscited by 72
  37. 2022Stitch it in Time: GAN-Based Facial Editing of Real Videoscited by 68
  38. 2020Compositional Learning of Image-Text Query for Image Retrievalcited by 57

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 30; 2024: 27; 2025: 19; 2026: 14.

95 papers.

Sort:

By area:

  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  2. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  3. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  4. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  5. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  6. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  7. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  8. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  9. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 155
  10. 2023A Neural Space-Time Representation for Text-to-Image Personalizationcited by 123
  11. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  12. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
38 more
  1. 2023Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Modelscited by 85
  2. 2022PODA: Prompt-driven Zero-shot Domain Adaptationcited by 60
  3. 2023Continual Learning: Applications and the Road Forwardcited by 59
  4. 2023Diagnosing and Rectifying Vision Models using Languagecited by 58
  5. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  6. 2023Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrievalcited by 52
  7. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  8. 2024Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrievalcited by 44
  9. 2024Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrievalcited by 36
  10. 2023ReFACT: Updating Text-to-Image Models by Editing the Text Encodercited by 32
  11. 2023Zero-shot Composed Text-Image Retrievalcited by 31
  12. 2024iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrievalcited by 24
  13. 2023Tailored Visions: Enhancing Text-to-Image Generation with Personalized Prompt Rewritingcited by 23
  14. 2023Training-free Zero-shot Composed Image Retrieval with Local Concept Rerankingcited by 23
  15. 2025CoLLM: A Large Language Model for Composed Image Retrievalcited by 21
  16. 2023Language-Guided Audio-Visual Source Separation via Trimodal Consistencycited by 20
  17. 2023Mapping Memes to Words for Multimodal Hateful Meme Classificationcited by 19
  18. 2025Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrievalcited by 18
  19. 2025A Comprehensive Survey on Composed Image Retrievalcited by 17
  20. 2023Meta-Personalizing Vision-Language Models to Find Named Instances in Videocited by 16
  21. 2023From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Modelscited by 16
  22. 2023Controllable Textual Inversion for Personalized Text-to-Image Generationcited by 14
  23. 2023Pretrain like Your Inference: Masked Tuning Improves Zero-Shot Composed Image Retrievalcited by 14
  24. 2024Prompt Sliders for Fine-Grained Control, Editing and Erasing of Concepts in Diffusion Modelscited by 14
  25. 2025Personalized Vision via Visual In-Context Learningcited by 14
  26. 2022Continual Learning with Evolving Class Ontologiescited by 13
  27. 2024Uncovering the Text Embedding in Text-to-Image Diffusion Modelscited by 13
  28. 2024Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrievalcited by 13
  29. 2023On the Powerfulness of Textual Outlier Exposure for Visual OoD Detectioncited by 12
  30. 2024You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrievalcited by 12
  31. 2024Bridging Vision and Language Spaces with Assignment Predictioncited by 11
  32. 2023Cross Initialization for Personalized Text-to-Image Generationcited by 10
  33. 2025Training-Free Personalization via Retrieval and Reasoning on Fingerprintscited by 10
  34. 2024Diffusion Models For Multi-Modal Generative Modelingcited by 9
  35. 2024Personalized Representation from Personalized Generationcited by 9
  36. 2023Agile Modeling: From Concept to Classifier in Minutescited by 8
  37. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  38. 2024BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generationcited by 8

50 most cited of 95 All 95 (JSON)