# Prompt Amplification and Zero-Shot Late Fusion in Audio-Language Models for Speech Emotion Recognition

arXiv: https://arxiv.org/abs/2603.23057 · HTML: https://mlsanity.com/paper/2603.23057/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Builds on (most cited first, 22 of 22)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://mlsanity.com/paper/2507.08128/) (2025)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmark](https://mlsanity.com/paper/2406.07162/) (2024)
- [A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models](https://mlsanity.com/paper/2302.06235/) (2023)
- [The MSP-Podcast Corpus](https://mlsanity.com/paper/2509.09791/) (2025)
- [ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds](https://mlsanity.com/paper/2409.09213/) (2024)
- [ParaCLAP -- Towards a general language-audio model for computational paralinguistic tasks](https://mlsanity.com/paper/2406.07203/) (2024)
- [Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thought](https://mlsanity.com/paper/2502.18186/) (2025)
- [Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training](https://mlsanity.com/paper/2601.03065/) (2026)
- [Classification of Emotions and Evaluation of Customer Satisfaction from Speech in Real World Acoustic Environments](https://mlsanity.com/paper/2108.11981/) (2021)
- [A sound description: Exploring prompt templates and class descriptions to enhance zero-shot audio classification](https://mlsanity.com/paper/2409.13676/) (2024)
- [Self-FiLM: Conditioning GANs with self-supervised representations for bandwidth extension based speaker recognition](https://mlsanity.com/paper/2303.03657/) (2023)
- [VoxEmo: Benchmarking Speech Emotion Recognition with Speech LLMs](https://mlsanity.com/paper/2603.08936/) (2026)
- [ABHINAYA -- A System for Speech Emotion Recognition In Naturalistic Conditions Challenge](https://mlsanity.com/paper/2505.18217/) (2025)

## Abstract (arXiv)

Audio-Language Models (ALMs) are making strides in understanding speech and non-speech audio. However, domain-specialist Foundation Models (FMs) remain the best for closed-ended speech processing tasks such as Speech Emotion Recognition (SER). Using ALMs for Zero-shot SER is a popular choice, but their potential to work with specialists to achieve state-of-the-art (SOTA) performance remains unexplored. We propose ZS-Fuse, a late-fusion method that combines zero-shot emotion estimates from a dual-encoder ALM with specialist FMs. To handle ambiguity in emotions and sensitivity to prompt choice, 1) we use a simple prompt ensemble and 2) suggest a novel technique called prompt amplification, which repeats audio and text queries to discover stronger zero-shot capabilities. We demonstrate the efficacy of our technique by evaluating ZS-Fuse with three dual-encoder ALMs and two FMs, and report improvements over SOTA baselines, such as WavLM-Large, on three speech emotion recognition datasets.