# Learning Temporal Resolution in Spectrogram for Audio Classification

arXiv: https://arxiv.org/abs/2210.01719 · HTML: https://mlsanity.com/paper/2210.01719/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 59 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1709.01953/) (2017)
- [PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation](https://mlsanity.com/paper/2102.01243/) (2021)
- [LEAF: A Learnable Frontend for Audio Classification](https://mlsanity.com/paper/2101.08596/) (2021)
- [Learning Fast Sample Re-weighting Without Reward Data](https://mlsanity.com/paper/2109.03216/) (2021)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [Slow-Fast Auditory Streams For Audio Recognition](https://mlsanity.com/paper/2103.03516/) (2021)
- [Broadcasted Residual Learning for Efficient Keyword Spotting](https://mlsanity.com/paper/2106.04140/) (2021)
- [Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separation](https://mlsanity.com/paper/2109.05418/) (2021)
- [Trainable Frontend For Robust and Far-Field Keyword Spotting](https://mlsanity.com/paper/1607.05666/) (2016)
- [Learning strides in convolutional neural networks](https://mlsanity.com/paper/2202.01653/) (2022)
- [Learning Filterbanks from Raw Speech for Phone Recognition](https://mlsanity.com/paper/1711.01161/) (2017)
- [Interpretable Convolutional Filters with SincNet](https://mlsanity.com/paper/1811.09725/) (2018)
- [Comparison of Time-Frequency Representations for Environmental Sound Classification using Convolutional Neural Networks](https://mlsanity.com/paper/1706.07156/) (2017)
- [Channel-wise Subband Input for Better Voice and Accompaniment Separation on High Resolution Music](https://mlsanity.com/paper/2008.05216/) (2020)

## Built on by (most cited first, 4 of 4)

- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Play It Back: Iterative Attention for Audio Recognition](https://mlsanity.com/paper/2210.11328/) (2022)
- [Ontology-aware Learning and Evaluation for Audio Tagging](https://mlsanity.com/paper/2211.12195/) (2022)
- [Comparative Analysis of Image, Video, and Audio Classifiers for Automated News Video Segmentation](https://mlsanity.com/paper/2503.21848/) (2025)

## Abstract (arXiv)

The audio spectrogram is a time-frequency representation that has been widely used for audio classification. One of the key attributes of the audio spectrogram is the temporal resolution, which depends on the hop size used in the Short-Time Fourier Transform (STFT). Previous works generally assume the hop size should be a constant value (e.g., 10 ms). However, a fixed temporal resolution is not always optimal for different types of sound. The temporal resolution affects not only classification accuracy but also computational cost. This paper proposes a novel method, DiffRes, that enables differentiable temporal resolution modeling for audio classification. Given a spectrogram calculated with a fixed hop size, DiffRes merges non-essential time frames while preserving important frames. DiffRes acts as a "drop-in" module between an audio spectrogram and a classifier and can be jointly optimized with the classification task. We evaluate DiffRes on five audio classification tasks, using mel-spectrograms as the acoustic features, followed by off-the-shelf classifier backbones. Compared with previous methods using the fixed temporal resolution, the DiffRes-based method can achieve the equivalent or better classification accuracy with at least 25% computational cost reduction. We further show that DiffRes can improve classification accuracy by increasing the temporal resolution of input acoustic features, without adding to the computational cost.