mlsanityData to 14 Oct 2026

ViP: A Differentially Private Foundation Model for Computer Vision

arXiv 2306.08842 ↗First posted 15 Jun 2023cs.CV cs.CR cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence (AI) has seen a tremendous surge in capabilities thanks to the use of foundation models trained on internet-scale data. On the flip side, the uncurated nature of internet-scale data also poses significant privacy and legal risks, as they often contain personal information or copyrighted material that should not be trained on without permission. In this work, we propose as a mitigation measure a recipe to train foundation vision models with differential privacy (DP) guarantee. We identify masked autoencoders as a suitable learning algorithm that aligns well with DP-SGD, and train ViP -- a Vision transformer with differential Privacy -- under a strict privacy budget of ε=8 on the LAION400M dataset. We evaluate the quality of representation learned by ViP using standard downstream vision tasks; in particular, ViP achieves a (non-private) linear probing accuracy of 55.7% on ImageNet, comparable to that of end-to-end trained AlexNet (trained and evaluated on ImageNet). Our result suggests that scaling to internet-scale data can be practical for private learning. Code is available at urlhttps://github.com/facebookresearch/ViP-MAE.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 49 of its 66 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

49 of 66 references.

References · 49

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2017Mask R-CNNcited by 7,277
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
37 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  3. 2016Deep Learning with Differential Privacycited by 2,728
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  10. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  13. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  14. 2017Renyi Differential Privacycited by 650
  15. 2017Large Batch Training of Convolutional Networkscited by 535
  16. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 465
  17. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  18. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  19. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  20. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  21. 2018Rethinking ImageNet Pre-trainingcited by 377
  22. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  23. 2023A Cookbook of Self-Supervised Learningcited by 250
  24. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  25. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  26. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 204
  27. 2021Large-Scale Differentially Private BERTcited by 118
  28. 2023Foundation Models and Fair Usecited by 116
  29. 2021Pre-training without Natural Imagescited by 100
  30. 2020BYOL works even without batch statisticscited by 95
  31. 2022Vision Transformers provably learn spatial structurecited by 93
  32. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  33. 2019Hypothesis Testing Interpretations and Renyi Differential Privacycited by 85
  34. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 52
  35. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 48
  36. 2022Procedural Image Programs for Representation Learningcited by 28
  37. 2023Do SSL Models Have D'ej`a Vu? A Case of Unintended Memorization in Self-supervised Learningcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 5; 2026: 1.

17 papers.

Sort:

By area:

  1. 2025Embodied AI Agents: Modeling the Worldcited by 61
  2. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  3. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  4. 2024Localizing Memorization in SSL Vision Encoderscited by 9
  5. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  6. 2024Differentially Private Representation Learning via Image Captioningcited by 5
  7. 2024Delving into Differentially Private Transformercited by 5
  8. 2024DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuningcited by 4
  9. 2024Implicit Bias in Noisy-SGD: With Applications to Differentially Private Trainingcited by 2
  10. 2025Machine Learning with Privacy for Protected Attributescited by 2
  11. 2025Private Zeroth-Order Optimization with Public Datacited by 2
  12. 2024Leveraging Programmatically Generated Synthetic Data for Differentially Private Diffusion Trainingcited by 1
5 more
  1. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  2. 2023Differentially Private Optimization for Non-Decomposable Objective Functionscited by 0
  3. 2024Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddingscited by 0
  4. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  5. 2026Lap2: Revisiting Laplace DP-SGD for High Dimensions via Majorization Theorycited by 0