# Hold Me Tight: Stable Encoder-Decoder Design for Speech Enhancement

arXiv: https://arxiv.org/abs/2408.17358 · HTML: https://mlsanity.com/paper/2408.17358/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 30 Aug 2024.

## Builds on (most cited first, 13 of 13)

- [SDR - half-baked or well done?](https://mlsanity.com/paper/1811.02508/) (2018)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [LEAF: A Learnable Frontend for Audio Classification](https://mlsanity.com/paper/2101.08596/) (2021)
- [Data augmentation and loss normalization for deep noise suppression](https://mlsanity.com/paper/2008.06412/) (2020)
- [Weighted Speech Distortion Losses for Neural-network-based Real-time Speech Enhancement](https://mlsanity.com/paper/2001.10601/) (2020)
- [Demystifying TasNet: A Dissecting Approach](https://mlsanity.com/paper/1911.08895/) (2019)
- [A Multi-Phase Gammatone Filterbank for Speech Separation via TasNet](https://mlsanity.com/paper/1910.11615/) (2019)
- [On Architectures and Training for Raw Waveform Feature Extraction in ASR](https://mlsanity.com/paper/2104.04298/) (2021)
- [Real-time Monaural Speech Enhancement With Short-time Discrete Cosine Transform](https://mlsanity.com/paper/2102.04629/) (2021)
- [Fitting Auditory Filterbanks with Multiresolution Neural Networks](https://mlsanity.com/paper/2307.13821/) (2023)
- [Instabilities in Convnets for Raw Audio](https://mlsanity.com/paper/2309.05855/) (2023)

## Built on by (most cited first, 2 of 2)

- [ISAC: An Invertible and Stable Auditory Filter Bank with Customizable Kernels for ML Integration](https://mlsanity.com/paper/2505.07709/) (2025)
- [Aliasing in Convnets: A Frame-Theoretic Perspective](https://mlsanity.com/paper/2507.06152/) (2025)

## Abstract (arXiv)

Convolutional layers with 1-D filters are often used as frontend to encode audio signals. Unlike fixed time-frequency representations, they can adapt to the local characteristics of input data. However, 1-D filters on raw audio are hard to train and often suffer from instabilities. In this paper, we address these problems with hybrid solutions, i.e., combining theory-driven and data-driven approaches. First, we preprocess the audio signals via a auditory filterbank, guaranteeing good frequency localization for the learned encoder. Second, we use results from frame theory to define an unsupervised learning objective that encourages energy conservation and perfect reconstruction. Third, we adapt mixed compressed spectral norms as learning objectives to the encoder coefficients. Using these solutions in a low-complexity encoder-mask-decoder model significantly improves the perceptual evaluation of speech quality (PESQ) in speech enhancement.