mlsanityData to 14 Oct 2026

Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Training

arXiv 2306.08173 ↗First posted 13 Jun 2023cs.LG cs.CR cs.ITcited by 7

Not on the map. Why?

Abstract (arXiv)

The surge in multimodal AI's success has sparked concerns over data privacy in vision-and-language tasks. While CLIP has revolutionized multimodal learning through joint training on images and text, its potential to unintentionally disclose sensitive information necessitates the integration of privacy-preserving mechanisms. We introduce a differentially private adaptation of the Contrastive Language-Image Pretraining (CLIP) model that effectively addresses privacy concerns while retaining accuracy. Our proposed method, Dp-CLIP, is rigorously evaluated on benchmark datasets encompassing diverse vision-and-language tasks such as image classification and visual question answering. We demonstrate that our approach retains performance on par with the standard non-private CLIP model. Furthermore, we analyze our proposed algorithm under linear representation settings. We derive the convergence rate of our algorithm and show a trade-off between utility and privacy when gradients are clipped per-batch and the loss function does not satisfy smoothness conditions assumed in the literature for the analysis of DP-SGD.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 61 of its 98 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

61 of 98 references.

References · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  7. 2016Deep Learning with Differential Privacycited by 2,728
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  12. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
38 more
  1. 2023Extracting Training Data from Diffusion Modelscited by 732
  2. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  3. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  4. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  5. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  6. 2022Multimodal Learning with Transformers: A Surveycited by 400
  7. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  8. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  9. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  10. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  11. 2019Private Stochastic Convex Optimization with Optimal Ratescited by 186
  12. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  13. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 160
  14. 2020Private Stochastic Convex Optimization: Optimal Rates in Linear Timecited by 153
  15. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
  16. 2018Differentially Private Empirical Risk Minimization Revisited: Faster and More Generalcited by 128
  17. 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 122
  18. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  19. 2019Deep Learning with Gaussian Differential Privacycited by 114
  20. 2020Captioning Images Taken by People Who Are Blindcited by 110
  21. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  22. 2023On Provable Copyright Protection for Generative Modelscited by 98
  23. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 88
  24. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
  25. 2018A General Approach to Adding Differential Privacy to Iterative Training Procedurescited by 84
  26. 2021EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learningcited by 73
  27. 2018Triplet-Center Loss for Multi-View 3D Object Retrievalcited by 69
  28. 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57
  29. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  30. 2021Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergencecited by 54
  31. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53
  32. 2020On Differentially Private Stochastic Convex Optimization with Heavy-tailed Datacited by 51
  33. 2023Understanding Multimodal Contrastive Learning and Incorporating Unpaired Datacited by 43
  34. 2021Quantifying and Mitigating Privacy Risks of Contrastive Learningcited by 36
  35. 2021On the Convergence and Calibration of Deep Learning with Differential Privacycited by 36
  36. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 36
  37. 2018Heteroskedastic PCA: Algorithm, Optimality, and Applicationscited by 34
  38. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 1.

7 papers.

  1. 2024A Survey on Safe Multi-Modal Learning Systemcited by 15
  2. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  3. 2024TUNI: A Textual Unimodal Detector for Identity Inference in CLIP Modelscited by 8
  4. 2025Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safetycited by 7
  5. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Modelscited by 4
  6. 2024Differentially Private Multimodal Laplacian Dropout (DP-MLD) for EEG Representative Learningcited by 1
  7. 2026TrustCLIP: Learning Private Visual Features via Adversarial Reconstructioncited by 0