# Spectrogram features for audio and speech analysis

arXiv: https://arxiv.org/abs/2603.14917 · HTML: https://mlsanity.com/paper/2603.14917/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 52 of its 159 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 30 of 52)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Deep clustering: Discriminative embeddings for segmentation and separation](https://mlsanity.com/paper/1508.04306/) (2015)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [Emotion Recognition from Speech Using Wav2vec 2.0 Embeddings](https://mlsanity.com/paper/2104.03502/) (2021)
- [Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks](https://mlsanity.com/paper/1807.00129/) (2018)
- [Computational bioacoustics with deep learning: a review and roadmap](https://mlsanity.com/paper/2112.06725/) (2021)
- [LEAF: A Learnable Frontend for Audio Classification](https://mlsanity.com/paper/2101.08596/) (2021)
- [CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization](https://mlsanity.com/paper/2004.15004/) (2020)
- [Sound Event Detection: A Tutorial](https://mlsanity.com/paper/2107.05463/) (2021)
- [Description and Discussion on DCASE2020 Challenge Task2: Unsupervised Anomalous Sound Detection for Machine Condition Monitoring](https://mlsanity.com/paper/2006.05822/) (2020)
- [Audio Transformers](https://mlsanity.com/paper/2105.00335/) (2021)
- [Anomalous sound detection based on interpolation deep neural network](https://mlsanity.com/paper/2005.09234/) (2020)
- [Trainable Frontend For Robust and Far-Field Keyword Spotting](https://mlsanity.com/paper/1607.05666/) (2016)
- [RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification](https://mlsanity.com/paper/1904.08104/) (2019)
- [Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveforms](https://mlsanity.com/paper/1703.01789/) (2017)
- [Improved Language Identification Through Cross-Lingual Self-Supervised Learning](https://mlsanity.com/paper/2107.04082/) (2021)
- [Mixup-Based Acoustic Scene Classification Using Multi-Channel Convolutional Neural Network](https://mlsanity.com/paper/1805.07319/) (2018)
- [Lung Sound Classification Using Co-tuning and Stochastic Normalization](https://mlsanity.com/paper/2108.01991/) (2021)
- [Anomalous Sound Detection using Spectral-Temporal Information Fusion](https://mlsanity.com/paper/2201.05510/) (2022)
- [AnoPatch: Towards Better Consistency in Machine Anomalous Sound Detection](https://mlsanity.com/paper/2406.11364/) (2024)

## Abstract (arXiv)

Spectrogram-based representations have grown to dominate the feature space for deep learning audio analysis systems, and are often adopted for speech analysis also. Initially, the primary motivator for spectrogram-based representations was their ability to present sound as a two dimensional signal in the time-frequency plane, which not only provides an interpretable physical basis for analysing sound, but also unlocks the use of a wide range of machine learning techniques such as convolutional neural networks, that had been developed for image processing. A spectrogram is a matrix characterised by the resolution and span of its two dimensions, as well as by the representation and scaling of each element. Many possibilities for these three characteristics have been explored by researchers across numerous application areas, with different settings showing affinity for various tasks. This paper reviews the use of spectrogram-based representations and surveys the state-of-the-art to question how front-end feature representation choice allies with back-end classifier architecture for different tasks.