mlsanityData to 7 Oct 2026

Understanding Visual Concepts Across Models

arXiv 2406.07506 ↗First posted 11 Jun 2024cs.CV cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

Large multimodal models such as Stable Diffusion can generate, detect, and classify new visual concepts after fine-tuning just a single word embedding. Do models learn similar words for the same concepts (i.e. <orange-cat> = orange + cat)? We conduct a large-scale analysis on three state-of-the-art models in text-to-image generation, open-set object detection, and zero-shot classification, and find that new word embeddings are model-specific and non-transferable. Across 4,800 new embeddings trained for 40 diverse visual concepts on four standard datasets, we find perturbations within an ε-ball to any prior embedding that generate, detect, and classify an arbitrary concept. When these new embeddings are spliced into new models, fine-tuning that targets the original model is lost. We show popular soft prompt-tuning approaches find these perturbative solutions when applied to visual concept learning tasks, and embeddings for visual concepts are not transferable. Code for reproducing our work is available at: https://visual-words.github.io.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 49 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

41 of 49 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
29 more
  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  10. 2016Adversarial examples in the physical worldcited by 1,762
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  13. 2023Segment Everything Everywhere All at Oncecited by 546
  14. 2017Adversarial Patchcited by 514
  15. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  16. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  17. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  18. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  19. 2023Your Diffusion Model is Secretly a Zero-Shot Classifiercited by 301
  20. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  21. 2018Adversarial Logit Pairingcited by 224
  22. 2022Unified Contrastive Learning in Image-Text-Label Spacecited by 194
  23. 2019Adversarial Robustness through Local Linearizationcited by 146
  24. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 140
  25. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  26. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  27. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  28. 2023Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Modelscited by 7
  29. 2023Continuous Prompt Generation from Linear Combination of Discrete Prompt Embeddingscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023The Hidden Language of Diffusion Models
  2. 2023ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models
  3. 2023Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else
  4. 2026Show Me Examples: Inferring Visual Concepts from Image Sets
  5. 2025SemPT: Semantic Prompt Tuning for Vision-Language Models
  6. 2025Bridging the gap to real-world language-grounded visual concept learning
  7. 2026Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
  8. 2024CusConcept: Customized Visual Concept Decomposition with Diffusion Models