mlsanityData to 7 Oct 2026

Differentially Private Representation Learning via Image Captioning

arXiv 2403.02506 ↗First posted 4 Mar 2024cs.CV cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Differentially private (DP) machine learning is considered the gold-standard solution for training a model from sensitive data while still preserving privacy. However, a major barrier to achieving this ideal is its sub-optimal privacy-accuracy trade-off, which is particularly visible in DP representation learning. Specifically, it has been shown that under modest privacy budgets, most models learn representations that are not significantly better than hand-crafted features. In this work, we show that effective DP representation learning can be done via image captioning and scaling up to internet-scale multimodal datasets. Through a series of engineering tricks, we successfully train a DP image captioner (DP-Cap) on a 233M subset of LAION-2B from scratch using a reasonable amount of computation, and obtaining unprecedented high-quality image features that can be used in a variety of downstream vision and vision-language tasks. For example, under a privacy budget of varepsilon=8 for the LAION dataset, a linear classifier trained on top of learned DP-Cap features attains 65.8% accuracy on ImageNet-1K, considerably improving the previous SOTA of 56.5%.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 57 of its 72 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

57 of 72 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
38 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2021Membership Inference Attacks From First Principlescited by 839
  6. 2017Renyi Differential Privacycited by 643
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  9. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  10. 2022DeiT III: Revenge of the ViTcited by 417
  11. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  12. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  13. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  14. 2016Concentrated Differential Privacy: Simplifications, Extensions, and Lower Boundscited by 339
  15. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  16. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  17. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  18. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  19. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  20. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  21. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  22. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  23. 2021Numerical Composition of Differential Privacycited by 174
  24. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  25. 2022Reconstructing Training Data with Informed Adversariescited by 143
  26. 2016Concentrated Differential Privacycited by 139
  27. 2020Learning Visual Representations with Caption Annotationscited by 123
  28. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  29. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  30. 2019Hypothesis Testing Interpretations and Renyi Differential Privacycited by 84
  31. 2021A Study of Face Obfuscation in ImageNetcited by 83
  32. 2018Private Selection from Private Candidatescited by 75
  33. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  34. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 51
  35. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  36. 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 46
  37. 2022Bounding Training Data Reconstruction in Private (Deep) Learningcited by 43
  38. 2022Procedural Image Programs for Representation Learningcited by 28

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2025Private Zeroth-Order Optimization with Public Datacited by 2
  2. 2024Differentially Private Multimodal Laplacian Dropout (DP-MLD) for EEG Representative Learningcited by 1
  3. 2024Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddingscited by 0
  4. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  5. 2026From Privacy to Generalization: Linear Max-Information Bounds for Differentially Private Learning Algorithmscited by 0