mlsanityData to 14 Oct 2026

D'ej`a Vu Memorization in Vision-Language Models

arXiv 2402.02103 ↗First posted 3 Feb 2024cs.CV cs.LGcited by 9

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) have emerged as the state-of-the-art representation learning solution, with myriads of downstream applications such as image classification, retrieval and generation. A natural question is whether these models memorize their training data, which also has implications for generalization. We propose a new method for measuring memorization in VLMs, which we call d'ej`a vu memorization. For VLMs trained on image-caption pairs, we show that the model indeed retains information about individual objects in the training images beyond what can be inferred from correlations or the image caption. We evaluate d'ej`a vu memorization at both sample and population level, and show that it is significant for OpenCLIP trained on as many as 50M image-caption pairs. Finally, we show that text randomization considerably mitigates memorization while only moderately impacting the model's downstream task performance.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 45 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

28 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  12. 2023Extracting Training Data from Diffusion Modelscited by 732
15 more
  1. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  3. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  4. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  5. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 424
  6. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  7. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  8. 2023Demystifying CLIP Datacited by 270
  9. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91
  10. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 90
  11. 2021A Study of Face Obfuscation in ImageNetcited by 83
  12. 2022Are Attribute Inference Attacks Just Imputation?cited by 49
  13. 2023Do SSL Models Have D'ej`a Vu? A Case of Unintended Memorization in Self-supervised Learningcited by 19
  14. 2018D'ej`a Vu: an empirical evaluation of the memorization properties of ConvNetscited by 10
  15. 2022Combing for Credentials: Active Pattern Extraction from Smart Replycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 2.

9 papers.

  1. 2024An Introduction to Vision-Language Modelingcited by 178
  2. 2025Membership Inference Attacks Against Vision-Language Modelscited by 29
  3. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  4. 2024Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Modelscited by 15
  5. 2024Understanding Depth and Height Perception in Large Visual-Language Modelscited by 8
  6. 2025LeakyCLIP: Extracting Training Data from CLIPcited by 4
  7. 2025DIS-CO: Discovering Copyrighted Content in VLMs Training Datacited by 1
  8. 2026MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learningcited by 0
  9. 2026COMPASS: Membership Inference Attack Against Vision Large Language Models via Adaptive Noise Calibrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Measuring D'ej`a vu Memorization Efficiently
  2. 2025Towards General Continuous Memory for Vision-Language Models
  3. 2025Quantifying Memorization and Parametric Response Rates in Retrieval-Augmented Vision-Language Models
  4. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
  5. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning
  6. 2023Text encoders bottleneck compositionality in contrastive vision-language models
  7. 2024DreamLIP: Language-Image Pre-training with Long Captions
  8. 2023Revisiting the Role of Language Priors in Vision-Language Models