# SemanticAC: Semantics-Assisted Framework for Audio Classification

arXiv: https://arxiv.org/abs/2302.05940 · HTML: https://mlsanity.com/paper/2302.05940/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 12 Feb 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Audio-Visual Instance Discrimination with Cross-Modal Agreement](https://mlsanity.com/paper/2004.12943/) (2020)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [Rethinking CNN Models for Audio Classification](https://mlsanity.com/paper/2007.11154/) (2020)
- [ESResNe(X)t-fbsp: Learning Robust Time-Frequency Transformation of Audio](https://mlsanity.com/paper/2104.11587/) (2021)
- [SimVTP: Simple Video Text Pre-training with Masked Autoencoders](https://mlsanity.com/paper/2212.03490/) (2022)
- [End-to-End Environmental Sound Classification using a 1D Convolutional Neural Network](https://mlsanity.com/paper/1904.08990/) (2019)
- [SepTr: Separable Transformer for Audio Spectrogram Processing](https://mlsanity.com/paper/2203.09581/) (2022)
- [Keep off the Grass: Permissible Driving Routes from Radar with Weak Audio Supervision](https://mlsanity.com/paper/2005.05175/) (2020)
- [Speaker Reinforcement Using Target Source Extraction for Robust Automatic Speech Recognition](https://mlsanity.com/paper/2205.04433/) (2022)

## Built on by (most cited first, 7 of 7)

- [SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation](https://mlsanity.com/paper/2305.17011/) (2023)
- [Strategic Preys Make Acute Predators: Enhancing Camouflaged Object Detectors by Generating Camouflaged Objects](https://mlsanity.com/paper/2308.03166/) (2023)
- [Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation](https://mlsanity.com/paper/2509.16630/) (2025)
- [UniQA: Unified Vision-Language Pre-training for Image Quality and Aesthetic Assessment](https://mlsanity.com/paper/2406.01069/) (2024)
- [Efficient Prompt Tuning of Large Vision-Language Model for Fine-Grained Ship Classification](https://mlsanity.com/paper/2403.08271/) (2024)
- [Gamma: Toward Generic Image Assessment with Mixture of Assessment Experts](https://mlsanity.com/paper/2503.06678/) (2025)
- [Video Object Segmentation with Dynamic Query Modulation](https://mlsanity.com/paper/2403.11529/) (2024)

## Abstract (arXiv)

In this paper, we propose SemanticAC, a semantics-assisted framework for Audio Classification to better leverage the semantic information. Unlike conventional audio classification methods that treat class labels as discrete vectors, we employ a language model to extract abundant semantics from labels and optimize the semantic consistency between audio signals and their labels. We verify that simple textual information from labels and advanced pretraining models enable more abundant semantic supervision for better performance. Specifically, we design a text encoder to capture the semantic information from the text extension of labels. Then we map the audio signals to align with the semantics of corresponding class labels via an audio encoder and a similarity calculation module so as to enforce the semantic consistency. Extensive experiments on two audio datasets, ESC-50 and US8K demonstrate that our proposed method consistently outperforms the compared audio classification methods.