# LEAF: A Learnable Frontend for Audio Classification

arXiv: https://arxiv.org/abs/2101.08596 · HTML: https://mlsanity.com/paper/2101.08596/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 61 references that are in the map.
- First posted on arXiv 21 Jan 2021.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation](https://mlsanity.com/paper/1910.06379/) (2019)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures](https://mlsanity.com/paper/1911.08460/) (2019)
- [Deep Scattering Spectrum](https://mlsanity.com/paper/1304.6763/) (2013)
- [Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge](https://mlsanity.com/paper/1807.05812/) (2018)
- [Trainable Frontend For Robust and Far-Field Keyword Spotting](https://mlsanity.com/paper/1607.05666/) (2016)
- [Learning Filterbanks from Raw Speech for Phone Recognition](https://mlsanity.com/paper/1711.01161/) (2017)
- [Filterbank design for end-to-end speech separation](https://mlsanity.com/paper/1910.10400/) (2019)
- [End-to-End Speech Recognition From the Raw Waveform](https://mlsanity.com/paper/1806.07098/) (2018)
- [Estimating Phoneme Class Conditional Probabilities from Raw Speech Signal using Convolutional Neural Networks](https://mlsanity.com/paper/1304.1018/) (2013)
- [CGCNN: Complex Gabor Convolutional Neural Network on raw speech](https://mlsanity.com/paper/2002.04569/) (2020)
- [Now Playing: Continuous low-power music recognition](https://mlsanity.com/paper/1711.10958/) (2017)
- [Multiclass Language Identification using Deep Learning on Spectral Images of Audio Signals](https://mlsanity.com/paper/1905.04348/) (2019)
- [Reducing Bias in Production Speech Models](https://mlsanity.com/paper/1705.04400/) (2017)
- [Attention-based Wav2Text with Feature Transfer Learning](https://mlsanity.com/paper/1709.07814/) (2017)

## Built on by (most cited first, 30 of 69)

- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [V2Meow: Meowing to the Visual Beat via Video-to-Music Generation](https://mlsanity.com/paper/2305.06594/) (2023)
- [ALLM4ADD: Unlocking the Capabilities of Audio Large Language Models for Audio Deepfake Detection](https://mlsanity.com/paper/2505.11079/) (2025)
- [Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy](https://mlsanity.com/paper/2401.00430/) (2023)
- [VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videos](https://mlsanity.com/paper/2409.07450/) (2024)
- [In Search for a Generalizable Method for Source Free Domain Adaptation](https://mlsanity.com/paper/2302.06658/) (2023)
- [HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition](https://mlsanity.com/paper/2304.06910/) (2023)
- [Brain2Music: Reconstructing Music from Human Brain Activity](https://mlsanity.com/paper/2307.11078/) (2023)
- [Simple Pooling Front-ends For Efficient Audio Classification](https://mlsanity.com/paper/2210.00943/) (2022)
- [AASIST3: KAN-Enhanced AASIST Speech Deepfake Detection using SSL Features and Additional Regularization for the ASVspoof 2024 Challenge](https://mlsanity.com/paper/2408.17352/) (2024)
- [FastAudio: A Learnable Audio Front-End for Spoof Speech Detection](https://mlsanity.com/paper/2109.02774/) (2021)
- [EfficientLEAF: A Faster LEarnable Audio Frontend of Questionable Use](https://mlsanity.com/paper/2207.05508/) (2022)
- [Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations](https://mlsanity.com/paper/2407.03495/) (2024)
- [Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks](https://mlsanity.com/paper/2107.00623/) (2021)
- [Comparing Supervised Models And Learned Speech Representations For Classifying Intelligibility Of Disordered Speech On Selected Phrases](https://mlsanity.com/paper/2107.03985/) (2021)
- [Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention](https://mlsanity.com/paper/2605.21842/) (2026)
- [Alzheimers Dementia Detection using Acoustic & Linguistic features and Pre-Trained BERT](https://mlsanity.com/paper/2109.11010/) (2021)
- [Learning Temporal Resolution in Spectrogram for Audio Classification](https://mlsanity.com/paper/2210.01719/) (2022)
- [Fitting Auditory Filterbanks with Multiresolution Neural Networks](https://mlsanity.com/paper/2307.13821/) (2023)
- [Comparative Analysis of the wav2vec 2.0 Feature Extractor](https://mlsanity.com/paper/2308.04286/) (2023)
- [A Comprehensive Survey on Generative AI for Video-to-Music Generation](https://mlsanity.com/paper/2502.12489/) (2025)
- [Instabilities in Convnets for Raw Audio](https://mlsanity.com/paper/2309.05855/) (2023)
- [TACNET: Temporal Audio Source Counting Network](https://mlsanity.com/paper/2311.02369/) (2023)
- [Pre-training Feature Guided Diffusion Model for Speech Enhancement](https://mlsanity.com/paper/2406.07646/) (2024)
- [Hold Me Tight: Stable Encoder-Decoder Design for Speech Enhancement](https://mlsanity.com/paper/2408.17358/) (2024)
- [Regularizing Learnable Feature Extraction for Automatic Speech Recognition](https://mlsanity.com/paper/2506.09804/) (2025)
- [DCER: Dual-Stage Compression and Energy-Based Reconstruction](https://mlsanity.com/paper/2602.04904/) (2026)
- [Chimpanzee voice prints? Insights from transfer learning experiments from human voices](https://mlsanity.com/paper/2112.08165/) (2021)

## Abstract (arXiv)

Mel-filterbanks are fixed, engineered audio features which emulate human perception and have been used through the history of audio understanding up to today. However, their undeniable qualities are counterbalanced by the fundamental limitations of handmade representations. In this work we show that we can train a single learnable frontend that outperforms mel-filterbanks on a wide range of audio signals, including speech, music, audio events and animal sounds, providing a general-purpose learned frontend for audio classification. To do so, we introduce a new principled, lightweight, fully learnable architecture that can be used as a drop-in replacement of mel-filterbanks. Our system learns all operations of audio features extraction, from filtering to pooling, compression and normalization, and can be integrated into any neural network at a negligible parameter cost. We perform multi-task training on eight diverse audio classification tasks, and show consistent improvements of our model over mel-filterbanks and previous learnable alternatives. Moreover, our system outperforms the current state-of-the-art learnable frontend on Audioset, with orders of magnitude fewer parameters.