# PALM: Few-Shot Prompt Learning for Audio Language Models

arXiv: https://arxiv.org/abs/2409.19806 · HTML: https://mlsanity.com/paper/2409.19806/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 29 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 12 of 12)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [Prompting Large Language Models with Speech Recognition Abilities](https://mlsanity.com/paper/2307.11795/) (2023)
- [PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation](https://mlsanity.com/paper/2102.01243/) (2021)
- [SpeechVerse: A Large-scale Generalizable Audio Language Model](https://mlsanity.com/paper/2405.08295/) (2024)
- [Audio-free Prompt Tuning for Language-Audio Models](https://mlsanity.com/paper/2309.08357/) (2023)
- [Domain Adaptation for Contrastive Audio-Language Models](https://mlsanity.com/paper/2402.09585/) (2024)

## Built on by (most cited first, 8 of 8)

- [Domain Adaptation Method and Modality Gap Impact in Audio-Text Models for Prototypical Sound Classification](https://mlsanity.com/paper/2506.04376/) (2025)
- [CLAP-S: Support Set Based Adaptation for Downstream Fiber-optic Acoustic Recognition](https://mlsanity.com/paper/2501.09877/) (2025)
- [Leveraging Prediction Entropy for Automatic Prompt Weighting in Zero-Shot Audio-Language Classification](https://mlsanity.com/paper/2601.05011/) (2026)
- [CALM: Class-Conditional Sparse Attention Vectors for Large Audio-Language Models](https://mlsanity.com/paper/2602.07077/) (2026)
- [TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control](https://mlsanity.com/paper/2603.09332/) (2026)
- [Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models](https://mlsanity.com/paper/2606.15751/) (2026)
- [ZEBRA: Zero-Shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization in Audio-Language Models](https://mlsanity.com/paper/2606.31587/) (2026)
- [SPECTRA: Subspace-Preserving Embedding Calibration, Transport, and Replay for Fully Few-Shot Class-Incremental Audio Classification](https://mlsanity.com/paper/2608.25054/) (2026)

## Abstract (arXiv)

Audio-Language Models (ALMs) have recently achieved remarkable success in zero-shot audio recognition tasks, which match features of audio waveforms with class-specific text prompt features, inspired by advancements in Vision-Language Models (VLMs). Given the sensitivity of zero-shot performance to the choice of hand-crafted text prompts, many prompt learning techniques have been developed for VLMs. We explore the efficacy of these approaches in ALMs and propose a novel method, Prompt Learning in Audio Language Models (PALM), which optimizes the feature space of the text encoder branch. Unlike existing methods that work in the input space, our approach results in greater training efficiency. We demonstrate the effectiveness of our approach on 11 audio recognition datasets, encompassing a variety of speech-processing tasks, and compare the results with three baselines in a few-shot learning setup. Our method is either on par with or outperforms other approaches while being computationally less demanding. Code is available at https://asif-hanif.github.io/palm/