# GAFX: A General Audio Feature eXtractor

arXiv: https://arxiv.org/abs/2207.09145 · HTML: https://mlsanity.com/paper/2207.09145/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 19 Jul 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [Music Source Separation in the Waveform Domain](https://mlsanity.com/paper/1911.13254/) (2019)
- [Hybrid Spectrogram and Waveform Source Separation](https://mlsanity.com/paper/2111.03600/) (2021)
- [LEAF: A Learnable Frontend for Audio Classification](https://mlsanity.com/paper/2101.08596/) (2021)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation](https://mlsanity.com/paper/2005.08575/) (2020)
- [Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed](https://mlsanity.com/paper/1909.01174/) (2019)
- [Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation](https://mlsanity.com/paper/2204.12260/) (2022)
- [Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training](https://mlsanity.com/paper/2204.12768/) (2022)
- [Rethinking CNN Models for Audio Classification](https://mlsanity.com/paper/2007.11154/) (2020)
- [ESResNet: Environmental Sound Classification Based on Visual Domain Models](https://mlsanity.com/paper/2004.07301/) (2020)
- [Sound Event Detection of Weakly Labelled Data with CNN-Transformer and Automatic Threshold Optimization](https://mlsanity.com/paper/1912.04761/) (2019)
- [Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events](https://mlsanity.com/paper/1912.02615/) (2019)

## Abstract (arXiv)

Most machine learning models for audio tasks are dealing with a handcrafted feature, the spectrogram. However, it is still unknown whether the spectrogram could be replaced with deep learning based features. In this paper, we answer this question by comparing the different learnable neural networks extracting features with a successful spectrogram model and proposed a General Audio Feature eXtractor (GAFX) based on a dual U-Net (GAFX-U), ResNet (GAFX-R), and Attention (GAFX-A) modules. We design experiments to evaluate this model on the music genre classification task on the GTZAN dataset and perform a detailed ablation study of different configurations of our framework and our model GAFX-U, following the Audio Spectrogram Transformer (AST) classifier achieves competitive performance.