# Exploring the Benefits of Visual Prompting in Differential Privacy

arXiv: https://arxiv.org/abs/2303.12247 · HTML: https://mlsanity.com/paper/2303.12247/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 45 references that are in the map.
- First posted on arXiv 22 Mar 2023.

## Builds on (most cited first, 30 of 37)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [Differentially Private Learning Needs Better Features (or Much More Data)](https://mlsanity.com/paper/2011.11660/) (2020)
- [FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning](https://mlsanity.com/paper/2205.07246/) (2022)
- [Unlocking High-Accuracy Differentially Private Image Classification through Scale](https://mlsanity.com/paper/2204.13650/) (2022)
- [Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning](https://mlsanity.com/paper/2101.04535/) (2021)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [USB: A Unified Semi-supervised Learning Benchmark for Classification](https://mlsanity.com/paper/2208.07204/) (2022)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [Large Scale Private Learning via Low-rank Reparametrization](https://mlsanity.com/paper/2106.09352/) (2021)
- [Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining](https://mlsanity.com/paper/2212.06470/) (2022)
- [Understanding and Improving Visual Prompting: A Label-Mapping Perspective](https://mlsanity.com/paper/2211.11635/) (2022)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)

## Built on by (most cited first, 17 of 17)

- [Privacy Preserving Prompt Engineering: A Survey](https://mlsanity.com/paper/2404.06001/) (2024)
- [REFINE: Inversion-Free Backdoor Defense via Model Reprogramming](https://mlsanity.com/paper/2502.18508/) (2025)
- [A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models](https://mlsanity.com/paper/2307.12980/) (2023)
- [DP-Mix: Mixup-based Data Augmentation for Differentially Private Learning](https://mlsanity.com/paper/2311.01295/) (2023)
- [Subgraph-level Universal Prompt Tuning](https://mlsanity.com/paper/2402.10380/) (2024)
- [Encapsulating Knowledge in One Prompt](https://mlsanity.com/paper/2407.11902/) (2024)
- [An Empirical Study of Federated Prompt Learning for Vision Language Model](https://mlsanity.com/paper/2505.23024/) (2025)
- [A2XP: Towards Private Domain Generalization](https://mlsanity.com/paper/2311.10339/) (2023)
- [Proactive Schemes: A Survey of Adversarial Attacks for Social Good](https://mlsanity.com/paper/2409.16491/) (2024)
- [Model Reprogramming Demystified: A Neural Tangent Kernel Perspective](https://mlsanity.com/paper/2506.00620/) (2025)
- [Prompting the Unseen: Detecting Hidden Backdoors in Black-Box Models](https://mlsanity.com/paper/2411.09540/) (2024)
- [SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generation](https://mlsanity.com/paper/2506.19360/) (2025)
- [Invisible Watermarks, Visible Gains: Steering Machine Unlearning with Bi-Level Watermarking Design](https://mlsanity.com/paper/2508.10065/) (2025)
- [Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers](https://mlsanity.com/paper/2309.06526/) (2023)
- [SecPE: Secure Prompt Ensembling for Private and Robust Large Language Models](https://mlsanity.com/paper/2502.00847/) (2025)
- [DP-GPL: Differentially Private Graph Prompt Learning](https://mlsanity.com/paper/2503.10544/) (2025)
- [VP-NTK: Exploring the Benefits of Visual Prompting in Differentially Private Data Synthesis](https://mlsanity.com/paper/2503.16195/) (2025)

## Abstract (arXiv)

Visual Prompting (VP) is an emerging and powerful technique that allows sample-efficient adaptation to downstream tasks by engineering a well-trained frozen source model. In this work, we explore the benefits of VP in constructing compelling neural network classifiers with differential privacy (DP). We explore and integrate VP into canonical DP training methods and demonstrate its simplicity and efficiency. In particular, we discover that VP in tandem with PATE, a state-of-the-art DP training method that leverages the knowledge transfer from an ensemble of teachers, achieves the state-of-the-art privacy-utility trade-off with minimum expenditure of privacy budget. Moreover, we conduct additional experiments on cross-domain image classification with a sufficient domain gap to further unveil the advantage of VP in DP. Lastly, we also conduct extensive ablation studies to validate the effectiveness and contribution of VP under DP consideration. Our code is available at (https://github.com/EzzzLi/Prompt-PATE).