# Sample-specific Masks for Visual Reprogramming-based Prompting

arXiv: https://arxiv.org/abs/2406.03150 · HTML: https://mlsanity.com/paper/2406.03150/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [Understanding and Improving Visual Prompting: A Label-Mapping Perspective](https://mlsanity.com/paper/2211.11635/) (2022)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning](https://mlsanity.com/paper/2307.13770/) (2023)
- [Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning](https://mlsanity.com/paper/2202.10629/) (2022)
- [Equivariant Adaptation of Large Pretrained Models](https://mlsanity.com/paper/2310.01647/) (2023)
- [Cross-modal Adversarial Reprogramming](https://mlsanity.com/paper/2102.07325/) (2021)
- [Fairness Reprogramming](https://mlsanity.com/paper/2209.10222/) (2022)
- [MeLo: Low-rank Adaptation is Better than Fine-tuning for Medical Image Diagnosis](https://mlsanity.com/paper/2311.08236/) (2023)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)
- [Watermarking for Out-of-distribution Detection](https://mlsanity.com/paper/2210.15198/) (2022)
- [From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition](https://mlsanity.com/paper/2301.07851/) (2023)
- [Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogramming](https://mlsanity.com/paper/2211.01317/) (2022)
- [TransHP: Image Classification with Hierarchical Prompting](https://mlsanity.com/paper/2304.06385/) (2023)
- [Efficient Equivariant Transfer Learning from Pretrained Models](https://mlsanity.com/paper/2305.09900/) (2023)
- [Three Towers: Flexible Contrastive Learning with Pretrained Image Models](https://mlsanity.com/paper/2305.16999/) (2023)
- [Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition](https://mlsanity.com/paper/2110.03894/) (2021)

## Built on by (most cited first, 18 of 18)

- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation](https://mlsanity.com/paper/2502.00896/) (2025)
- [Ranked Entropy Minimization for Continual Test-Time Adaptation](https://mlsanity.com/paper/2505.16441/) (2025)
- [Bayesian-guided Label Mapping for Visual Reprogramming](https://mlsanity.com/paper/2410.24018/) (2024)
- [Attribute-based Visual Reprogramming for Vision-Language Models](https://mlsanity.com/paper/2501.13982/) (2025)
- [Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts](https://mlsanity.com/paper/2506.01000/) (2025)
- [Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction](https://mlsanity.com/paper/2506.04650/) (2025)
- [Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations](https://mlsanity.com/paper/2506.02696/) (2025)
- [VirDA: Reusing Backbone for Unsupervised Domain Adaptation with Visual Reprogramming](https://mlsanity.com/paper/2510.01660/) (2025)
- [Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models](https://mlsanity.com/paper/2601.20419/) (2026)
- [Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation](https://mlsanity.com/paper/2510.07823/) (2025)
- [Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise](https://mlsanity.com/paper/2604.09532/) (2026)
- [BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogramming](https://mlsanity.com/paper/2606.24740/) (2026)
- [ES-VP : Energy-Shaped Dynamic Visual Prompting for Efficient Model Adaptation](https://mlsanity.com/paper/2608.21194/) (2026)
- [You Only Reprogram Once: Rethinking Prolonged Training for Visual Reprogramming](https://mlsanity.com/paper/2609.36661/) (2026)
- [Reprogramming Vision-Language Models via Structured Prompt Reparameterization](https://mlsanity.com/paper/2609.36680/) (2026)
- [WeatherBridge: Forecast-Aware Multi-Source Input Adaptation for Pretrained Weather Forecasting Models](https://mlsanity.com/paper/2610.14863/) (2026)

## Abstract (arXiv)

Visual reprogramming (VR) is a prompting technique that aims to re-purpose a pre-trained model (e.g., a classifier on ImageNet) to target tasks (e.g., medical data prediction) by learning a small-scale pattern added into input images instead of tuning considerable parameters within the model. The location of the pattern within input samples is usually determined by a pre-defined mask shared across all samples. In this paper, we show that the shared mask potentially limits VR's generalization and increases its approximation error due to the lack of sample-level adaptation. Motivated by this finding, we design a new framework for VR called sample-specific multi-channel masks (SMM). Specifically, SMM employs a lightweight ConvNet and patch-wise interpolation to generate sample-specific three-channel masks instead of a shared and pre-defined mask. Since we generate different masks for individual samples, SMM is theoretically shown to reduce approximation error for the target tasks compared with existing state-of-the-art VR methods. We also empirically demonstrate its performance gain on both ResNet and ViT. The success of SMM further highlights the broader applicability of VR in leveraging the latent knowledge of pre-trained models for various target tasks. Our code is available at https://github.com/tmlr-group/SMM.