# TSPE: Task-Specific Prompt Ensemble for Improved Zero-Shot Audio Classification

arXiv: https://arxiv.org/abs/2501.00398 · HTML: https://mlsanity.com/paper/2501.00398/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 31 Dec 2024.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2311.13165/) (2023)
- [A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models](https://mlsanity.com/paper/2110.08484/) (2021)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [A Survey of Multimodal Large Language Model from A Data-centric Perspective](https://mlsanity.com/paper/2405.16640/) (2024)
- [Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition](https://mlsanity.com/paper/2205.03433/) (2022)
- [CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models](https://mlsanity.com/paper/2310.08753/) (2023)
- [Grounding Multimodal Large Language Models in Actions](https://mlsanity.com/paper/2406.07904/) (2024)
- [CochlScene: Acquisition of acoustic scene data using crowdsourcing](https://mlsanity.com/paper/2211.02289/) (2022)
- [A Review of Multi-Modal Large Language and Vision Models](https://mlsanity.com/paper/2404.01322/) (2024)
- [Audio-free Prompt Tuning for Language-Audio Models](https://mlsanity.com/paper/2309.08357/) (2023)
- [Adapting Language-Audio Models as Few-Shot Audio Learners](https://mlsanity.com/paper/2305.17719/) (2023)

## Built on by (most cited first, 2 of 2)

- [Leveraging Prediction Entropy for Automatic Prompt Weighting in Zero-Shot Audio-Language Classification](https://mlsanity.com/paper/2601.05011/) (2026)
- [Drift-Augmented Scoring: Text-Derived Noise Robustness for Zero-Shot Audio-Language Classification](https://mlsanity.com/paper/2606.04844/) (2026)

## Abstract (arXiv)

Audio-language models (ALMs) excel in zero-shot audio classification, a task where models classify previously unseen audio clips at test time by leveraging descriptive natural language prompts. We introduce TSPE (Task-Specific Prompt Ensemble), a simple, training-free hard prompting method that boosts ALEs' zero-shot performance by customizing prompts for diverse audio classification tasks. Rather than using generic template-based prompts like "Sound of a car" we generate context-rich prompts, such as "Sound of a car coming from a tunnel". Specifically, we leverage label information to identify suitable sound attributes, such as "loud" and "feeble", and appropriate sound sources, such as "tunnel" and "street" and incorporate this information into the prompts used by Audio-Language Models (ALMs) for audio classification. Further, to enhance audio-text alignment, we perform prompt ensemble across TSPE-generated task-specific prompts. When evaluated on 12 diverse audio classification datasets, TSPE improves performance across ALMs by showing an absolute improvement of 1.23-16.36% over vanilla zero-shot evaluation.