# On the choice of the optimal temporal support for audio classification with Pre-trained embeddings

arXiv: https://arxiv.org/abs/2312.14005 · HTML: https://mlsanity.com/paper/2312.14005/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 21 of 21)

- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [Towards Learning Universal Audio Representations](https://mlsanity.com/paper/2111.12124/) (2021)
- [BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations](https://mlsanity.com/paper/2204.07402/) (2022)
- [Adaptive pooling operators for weakly labeled sound event detection](https://mlsanity.com/paper/1804.10070/) (2018)
- [Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study](https://mlsanity.com/paper/2303.06740/) (2023)
- [An Attention Mechanism for Musical Instrument Recognition](https://mlsanity.com/paper/1907.04294/) (2019)
- [Description and analysis of novelties introduced in DCASE Task 4 2022 on the baseline system](https://mlsanity.com/paper/2210.07856/) (2022)
- [An Attention-based Approach to Hierarchical Multi-label Music Instrument Classification](https://mlsanity.com/paper/2302.08136/) (2023)
- [Learning General Audio Representations with Large-Scale Training of Patchout Audio Transformers](https://mlsanity.com/paper/2211.13956/) (2022)

## Built on by (most cited first, 1 of 1)

- [Investigating Design Choices in Joint-Embedding Predictive Architectures for General Audio Representation Learning](https://mlsanity.com/paper/2405.08679/) (2024)

## Abstract (arXiv)

Current state-of-the-art audio analysis systems rely on pre-trained embedding models, often used off-the-shelf as (frozen) feature extractors. Choosing the best one for a set of tasks is the subject of many recent publications. However, one aspect often overlooked in these works is the influence of the duration of audio input considered to extract an embedding, which we refer to as Temporal Support (TS). In this work, we study the influence of the TS for well-established or emerging pre-trained embeddings, chosen to represent different types of architectures and learning paradigms. We conduct this evaluation using both musical instrument and environmental sound datasets, namely OpenMIC, TAU Urban Acoustic Scenes 2020 Mobile, and ESC-50. We especially highlight that Audio Spectrogram Transformer-based systems (PaSST and BEATs) remain effective with smaller TS, which therefore allows for a drastic reduction in memory and computational cost. Moreover, we show that by choosing the optimal TS we reach competitive results across all tasks. In particular, we improve the state-of-the-art results on OpenMIC, using BEATs and PaSST without any fine-tuning.