# Adapting a ConvNeXt model to audio classification on AudioSet

arXiv: https://arxiv.org/abs/2306.00830 · HTML: https://mlsanity.com/paper/2306.00830/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2023.

## Built on by (most cited first, 14 of 14)

- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [CED: Consistent ensemble distillation for audio tagging](https://mlsanity.com/paper/2308.11957/) (2023)
- [EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning](https://mlsanity.com/paper/2401.17690/) (2024)
- [DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning](https://mlsanity.com/paper/2410.09472/) (2024)
- [A Survey of Foundation Models for Music Understanding](https://mlsanity.com/paper/2409.09601/) (2024)
- [Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?](https://mlsanity.com/paper/2308.15090/) (2023)
- [Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models](https://mlsanity.com/paper/2310.15648/) (2023)
- [EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance](https://mlsanity.com/paper/2409.01201/) (2024)
- [Audio classification with Dilated Convolution with Learnable Spacings](https://mlsanity.com/paper/2309.13972/) (2023)
- [Expanding on EnCLAP with Auxiliary Retrieval Model for Automated Audio Captioning](https://mlsanity.com/paper/2409.01160/) (2024)
- [Machine listening in a neonatal intensive care unit](https://mlsanity.com/paper/2409.11439/) (2024)
- [Hierarchical Label Propagation: A Model-Size-Dependent Performance Booster for AudioSet Tagging](https://mlsanity.com/paper/2503.21826/) (2025)
- [Comprehensive Evaluation of CNN-Based Audio Tagging Models on Resource-Constrained Devices](https://mlsanity.com/paper/2509.14049/) (2025)
- [Shared Representation Learning for Reference-Guided Targeted Sound Detection](https://mlsanity.com/paper/2603.17025/) (2026)

## Abstract (arXiv)

In computer vision, convolutional neural networks (CNN) such as ConvNeXt, have been able to surpass state-of-the-art transformers, partly thanks to depthwise separable convolutions (DSC). DSC, as an approximation of the regular convolution, has made CNNs more efficient in time and memory complexity without deteriorating their accuracy, and sometimes even improving it. In this paper, we first implement DSC into the Pretrained Audio Neural Networks (PANN) family for audio classification on AudioSet, to show its benefits in terms of accuracy/model size trade-off. Second, we adapt the now famous ConvNeXt model to the same task. It rapidly overfits, so we report on techniques that improve the learning process. Our best ConvNeXt model reached 0.471 mean-average precision on AudioSet, which is better than or equivalent to recent large audio transformers, while using three times less parameters. We also achieved positive results in audio captioning and audio retrieval with this model. Our PyTorch source code and checkpoint models are available at https://github.com/topel/audioset-convnext-inf.