# PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation

arXiv: https://arxiv.org/abs/2102.01243 · HTML: https://mlsanity.com/paper/2102.01243/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 61 references that are in the map.
- First posted on arXiv 2 Feb 2021.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [An Unsupervised Autoregressive Model for Speech Representation Learning](https://mlsanity.com/paper/1904.03240/) (2019)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average](https://mlsanity.com/paper/1806.05594/) (2018)
- [Learning from Between-class Examples for Deep Sound Recognition](https://mlsanity.com/paper/1711.10282/) (2017)
- [Between-class Learning for Image Classification](https://mlsanity.com/paper/1711.10284/) (2017)
- [Rethinking CNN Models for Audio Classification](https://mlsanity.com/paper/2007.11154/) (2020)
- [Unsupervised Learning of Semantic Audio Representations](https://mlsanity.com/paper/1711.02209/) (2017)
- [A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling](https://mlsanity.com/paper/1810.09050/) (2018)
- [ESResNet: Environmental Sound Classification Based on Visual Domain Models](https://mlsanity.com/paper/2004.07301/) (2020)
- [Knowledge Transfer from Weakly Labeled Audio using Convolutional Neural Network for Sound Events and Scenes](https://mlsanity.com/paper/1711.01369/) (2017)
- [Weakly Labelled AudioSet Tagging with Attention Neural Networks](https://mlsanity.com/paper/1903.00765/) (2019)
- [Audio Set classification with attention model: A probabilistic perspective](https://mlsanity.com/paper/1711.00927/) (2017)
- [A Closer Look at Weak Label Learning for Audio Events](https://mlsanity.com/paper/1804.09288/) (2018)

## Built on by (most cited first, 30 of 82)

- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [Contrastive Audio-Visual Masked Autoencoder](https://mlsanity.com/paper/2210.07839/) (2022)
- [Joint Audio and Speech Understanding](https://mlsanity.com/paper/2309.14405/) (2023)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [Vision Transformers are Parameter-Efficient Audio-Visual Learners](https://mlsanity.com/paper/2212.07983/) (2022)
- [UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition](https://mlsanity.com/paper/2311.15599/) (2023)
- [Vocalsound: A Dataset for Improving Human Vocal Sounds Recognition](https://mlsanity.com/paper/2205.03433/) (2022)
- [A-JEPA: Joint-Embedding Predictive Architecture Can Listen](https://mlsanity.com/paper/2311.15830/) (2023)
- [EAT: Self-Supervised Pre-Training with Efficient Audio Transformer](https://mlsanity.com/paper/2401.03497/) (2024)
- [Audiovisual Masked Autoencoders](https://mlsanity.com/paper/2212.05922/) (2022)
- [ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound](https://mlsanity.com/paper/2204.02874/) (2022)
- [Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion](https://mlsanity.com/paper/2312.04466/) (2023)
- [Supervised and Unsupervised Learning of Audio Representations for Music Understanding](https://mlsanity.com/paper/2210.03799/) (2022)
- [XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning](https://mlsanity.com/paper/2211.13929/) (2022)
- [Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Data](https://mlsanity.com/paper/2112.07891/) (2021)
- [Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation](https://mlsanity.com/paper/2211.04772/) (2022)
- [Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks](https://mlsanity.com/paper/2306.04186/) (2023)
- [Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches](https://mlsanity.com/paper/2405.04771/) (2024)
- [UAVM: Towards Unifying Audio and Visual Models](https://mlsanity.com/paper/2208.00061/) (2022)
- [Multiscale Audio Spectrogram Transformer for Efficient Audio Classification](https://mlsanity.com/paper/2303.10757/) (2023)
- [Siamese Vision Transformers are Scalable Audio-visual Learners](https://mlsanity.com/paper/2403.19638/) (2024)
- [CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification](https://mlsanity.com/paper/2203.06760/) (2022)
- [Fourier Head: Helping Large Language Models Learn Complex Probability Distributions](https://mlsanity.com/paper/2410.22269/) (2024)
- [Simple Pooling Front-ends For Efficient Audio Classification](https://mlsanity.com/paper/2210.00943/) (2022)
- [PALM: Few-Shot Prompt Learning for Audio Language Models](https://mlsanity.com/paper/2409.19806/) (2024)

## Abstract (arXiv)

Audio tagging is an active research area and has a wide range of applications. Since the release of AudioSet, great progress has been made in advancing model performance, which mostly comes from the development of novel model architectures and attention modules. However, we find that appropriate training techniques are equally important for building audio tagging models with AudioSet, but have not received the attention they deserve. To fill the gap, in this work, we present PSLA, a collection of training techniques that can noticeably boost the model accuracy including ImageNet pretraining, balanced sampling, data augmentation, label enhancement, model aggregation and their design choices. By training an EfficientNet with these techniques, we obtain a single model (with 13.6M parameters) and an ensemble model that achieve mean average precision (mAP) scores of 0.444 and 0.474 on AudioSet, respectively, outperforming the previous best system of 0.439 with 81M parameters. In addition, our model also achieves a new state-of-the-art mAP of 0.567 on FSD50K.