# Training Spatial-Frequency Visual Prompts and Probabilistic Clusters for Accurate Black-Box Transfer Learning

arXiv: https://arxiv.org/abs/2408.07944 · HTML: https://mlsanity.com/paper/2408.07944/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 15 Aug 2024.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [A Fourier-based Framework for Domain Generalization](https://mlsanity.com/paper/2105.11120/) (2021)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning](https://mlsanity.com/paper/2006.06224/) (2020)
- [Unified Vision and Language Prompt Learning](https://mlsanity.com/paper/2210.07225/) (2022)
- [Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources](https://mlsanity.com/paper/2007.08714/) (2020)
- [Low Frequency Adversarial Perturbation](https://mlsanity.com/paper/1809.08758/) (2018)
- [BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning](https://mlsanity.com/paper/2303.14773/) (2023)
- [Phase Consistent Ecological Domain Adaptation](https://mlsanity.com/paper/2004.04923/) (2020)
- [Unleashing the Power of Visual Prompting At the Pixel Level](https://mlsanity.com/paper/2212.10556/) (2022)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)
- [Explicit Visual Prompting for Universal Foreground Segmentations](https://mlsanity.com/paper/2305.18476/) (2023)

## Built on by (most cited first, 1 of 1)

- [Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation](https://mlsanity.com/paper/2510.07823/) (2025)

## Abstract (arXiv)

Despite the growing prevalence of black-box pre-trained models (PTMs) such as prediction API services, there remains a significant challenge in directly applying general models to real-world scenarios due to the data distribution gap. Considering a data deficiency and constrained computational resource scenario, this paper proposes a novel parameter-efficient transfer learning framework for vision recognition models in the black-box setting. Our framework incorporates two novel training techniques. First, we align the input space (i.e., image) of PTMs to the target data distribution by generating visual prompts of spatial and frequency domain. Along with the novel spatial-frequency hybrid visual prompter, we design a novel training technique based on probabilistic clusters, which can enhance class separation in the output space (i.e., prediction probabilities). In experiments, our model demonstrates superior performance in a few-shot transfer learning setting across extensive visual recognition datasets, surpassing state-of-the-art baselines. Additionally, we show that the proposed method efficiently reduces computational costs for training and inference phases.