# Zero-Shot Audio Classification via Semantic Embeddings

arXiv: https://arxiv.org/abs/2011.12133 · HTML: https://mlsanity.com/paper/2011.12133/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2020.

## Built on by (most cited first, 21 of 21)

- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language](https://mlsanity.com/paper/2203.03598/) (2022)
- [EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge](https://mlsanity.com/paper/2311.10986/) (2023)
- [Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data](https://mlsanity.com/paper/2410.02056/) (2024)
- [Zero-Shot Multi-Label Topic Inference with Sentence Encoders](https://mlsanity.com/paper/2304.07382/) (2023)
- [Exploring Meta Information for Audio-based Zero-shot Bird Classification](https://mlsanity.com/paper/2309.08398/) (2023)
- [I can listen but cannot read: An evaluation of two-tower multimodal systems for instrument recognition](https://mlsanity.com/paper/2407.18058/) (2024)
- [Leveraging Foundation Models for Zero-Shot IoT Sensing](https://mlsanity.com/paper/2407.19893/) (2024)
- [Zero-Shot Audio Classification using Image Embeddings](https://mlsanity.com/paper/2206.04984/) (2022)
- [Improved Zero-Shot Audio Tagging & Classification with Patchout Spectrogram Transformers](https://mlsanity.com/paper/2208.11402/) (2022)
- [Multi-label Zero-Shot Audio Classification with Temporal Attention](https://mlsanity.com/paper/2409.00408/) (2024)
- [Zero-Shot Multi-Label Classification of Bangla Documents: Large Decoders Vs. Classic Encoders](https://mlsanity.com/paper/2503.02993/) (2025)
- [Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning](https://mlsanity.com/paper/2505.19938/) (2025)
- [GPLA-12: An Acoustic Signal Dataset of Gas Pipeline Leakage](https://mlsanity.com/paper/2106.10277/) (2021)
- [Generalized zero-shot audio-to-intent classification](https://mlsanity.com/paper/2311.02482/) (2023)
- [On Class Separability Pitfalls In Audio-Text Contrastive Zero-Shot Learning](https://mlsanity.com/paper/2408.13068/) (2024)
- [Climate Change & Computer Audition: A Call to Action and Overview on Audio Intelligence to Help Save the Planet](https://mlsanity.com/paper/2203.06064/) (2022)
- [A Benchmark of Generative Methods for Zero-Shot Environmental Sound Classification](https://mlsanity.com/paper/2412.03771/) (2024)
- [Learning Normal Patterns in Musical Loops](https://mlsanity.com/paper/2505.23784/) (2025)
- [Audio emotion recognition for atypical hearing](https://mlsanity.com/paper/2609.31168/) (2026)

## Abstract (arXiv)

In this paper, we study zero-shot learning in audio classification via semantic embeddings extracted from textual labels and sentence descriptions of sound classes. Our goal is to obtain a classifier that is capable of recognizing audio instances of sound classes that have no available training samples, but only semantic side information. We employ a bilinear compatibility framework to learn an acoustic-semantic projection between intermediate-level representations of audio instances and sound classes, i.e., acoustic embeddings and semantic embeddings. We use VGGish to extract deep acoustic embeddings from audio clips, and pre-trained language models (Word2Vec, GloVe, BERT) to generate either label embeddings from textual labels or sentence embeddings from sentence descriptions of sound classes. Audio classification is performed by a linear compatibility function that measures how compatible an acoustic embedding and a semantic embedding are. We evaluate the proposed method on a small balanced dataset ESC-50 and a large-scale unbalanced audio subset of AudioSet. The experimental results show that classification performance is significantly improved by involving sound classes that are semantically close to the test classes in training. Meanwhile, we demonstrate that both label embeddings and sentence embeddings are useful for zero-shot learning. Classification performance is improved by concatenating label/sentence embeddings generated with different language models. With their hybrid concatenations, the results are improved further.