# Generalizable Prompt Tuning for Audio-Language Models via Semantic Expansion

arXiv: https://arxiv.org/abs/2601.20867 · HTML: https://mlsanity.com/paper/2601.20867/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [Visual-Language Prompt Tuning with Knowledge-guided Context Optimization](https://mlsanity.com/paper/2303.13283/) (2023)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation](https://mlsanity.com/paper/2102.01243/) (2021)
- [Natural Language Supervision for General-Purpose Audio Representations](https://mlsanity.com/paper/2309.05767/) (2023)
- [Improving Generalization via Scalable Neighborhood Component Analysis](https://mlsanity.com/paper/1808.04699/) (2018)
- [Audio-free Prompt Tuning for Language-Audio Models](https://mlsanity.com/paper/2309.08357/) (2023)
- [What Variables Affect Out-of-Distribution Generalization in Pretrained Models?](https://mlsanity.com/paper/2405.15018/) (2024)
- [CLEP-DG: Contrastive Learning for Speech Emotion Domain Generalization via Soft Prompt Tuning](https://mlsanity.com/paper/2507.04048/) (2025)

## Abstract (arXiv)

Prompt tuning has achieved remarkable progress in vision-language models (VLMs) and is recently being adopted for audio-language models (ALMs). However, its generalization ability in ALMs remains largely underexplored. We observe that conventional prompt tuning for ALMs also suffers from the Base-New Tradeoff, and we identify that this issue stems from the disrupted semantic structure of the embedding space. To address this issue, we propose Semantically Expanded Prompt Tuning (SEPT)-a plug-and-play framework that explicitly regularizes the prompt embedding space by incorporating semantic neighbors generated by large language models. SEPT introduces a novel semantic expansion loss with margin constraints that promote intra-class compactness and inter-class separability, thereby enhancing the semantic structure of the prompt embedding space. For comprehensive evaluation, we establish the first benchmark setup for prompt generalization in ALMs, covering both base-to-new generalization and cross-dataset transferability. Extensive experiments demonstrate that SEPT consistently improves generalization performance across multiple prompt tuning baselines, while maintaining computational cost during inference.