# Audio-free Prompt Tuning for Language-Audio Models

arXiv: https://arxiv.org/abs/2309.08357 · HTML: https://mlsanity.com/paper/2309.08357/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2023.

## Built on by (most cited first, 13 of 13)

- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [PALM: Few-Shot Prompt Learning for Audio Language Models](https://mlsanity.com/paper/2409.19806/) (2024)
- [A sound description: Exploring prompt templates and class descriptions to enhance zero-shot audio classification](https://mlsanity.com/paper/2409.13676/) (2024)
- [Fine-Tuning Large Audio-Language Models with LoRA for Precise Temporal Localization of Prolonged Exposure Therapy Elements](https://mlsanity.com/paper/2506.09707/) (2025)
- [Text as Any-Modality for Zero-Shot Classification by Consistent Prompt Tuning](https://mlsanity.com/paper/2508.06382/) (2025)
- [TSPE: Task-Specific Prompt Ensemble for Improved Zero-Shot Audio Classification](https://mlsanity.com/paper/2501.00398/) (2024)
- [Multiple Consistency-guided Test-Time Adaptation for Contrastive Audio-Language Models with Unlabeled Audio](https://mlsanity.com/paper/2412.17306/) (2024)
- [CLAP-S: Support Set Based Adaptation for Downstream Fiber-optic Acoustic Recognition](https://mlsanity.com/paper/2501.09877/) (2025)
- [SPANER: Shared Prompt Aligner for Multimodal Semantic Representation](https://mlsanity.com/paper/2508.13387/) (2025)
- [EMO-TTA: Improving Test-Time Adaptation of Audio-Language Models for Speech Emotion Recognition](https://mlsanity.com/paper/2509.25495/) (2025)
- [TAIL: Text-Audio Incremental Learning](https://mlsanity.com/paper/2503.04258/) (2025)
- [Generalizable Prompt Tuning for Audio-Language Models via Semantic Expansion](https://mlsanity.com/paper/2601.20867/) (2026)
- [Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models](https://mlsanity.com/paper/2606.15751/) (2026)

## Abstract (arXiv)

Contrastive Language-Audio Pretraining (CLAP) is pre-trained to associate audio features with human language, making it a natural zero-shot classifier to recognize unseen sound categories. To adapt CLAP to downstream tasks, prior works inevitably require labeled domain audios, which limits their scalability under data scarcity and deprives them of the capability to detect novel classes as the original CLAP. In this work, by leveraging the modality alignment in CLAP, we propose an efficient audio-free prompt tuning scheme aimed at optimizing a few prompt tokens from texts instead of audios, which regularizes the model space to avoid overfitting the seen classes as well. Based on this, a multi-grained prompt design is further explored to fuse global and local information. Experiments on several tasks demonstrate that our approach can boost the CLAP and outperform other training methods on model performance and training efficiency. While conducting zero-shot inference on unseen categories, it still shows better transferability than the vanilla CLAP. Moreover, our method is flexible enough even if only knowing the downstream class names. The code will be released soon.