# PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech Enhancement

arXiv: https://arxiv.org/abs/2302.08095 · HTML: https://mlsanity.com/paper/2302.08095/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Builds on (most cited first, 19 of 19)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement](https://mlsanity.com/paper/2104.03538/) (2021)
- [MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement](https://mlsanity.com/paper/1905.04874/) (2019)
- [FullSubNet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement](https://mlsanity.com/paper/2010.15508/) (2020)
- [ICASSP 2022 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2202.13288/) (2022)
- [A scalable noisy speech dataset and online subjective test framework](https://mlsanity.com/paper/1909.08050/) (2019)
- [Phone-to-audio alignment without text: A Semi-supervised Approach](https://mlsanity.com/paper/2110.03876/) (2021)
- [A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences](https://mlsanity.com/paper/2001.04460/) (2020)
- [Convolutional-Recurrent Neural Networks for Speech Enhancement](https://mlsanity.com/paper/1805.00579/) (2018)
- [I'm Sorry for Your Loss: Spectrally-Based Audio Distances Are Bad at Pitch](https://mlsanity.com/paper/2012.04572/) (2020)
- [Speech Quality Assessment through MOS using Non-Matching References](https://mlsanity.com/paper/2206.12285/) (2022)
- [A Systematic Comparison of Phonetic Aware Techniques for Speech Enhancement](https://mlsanity.com/paper/2206.11000/) (2022)
- [Improving Speech Enhancement through Fine-Grained Speech Characteristics](https://mlsanity.com/paper/2207.00237/) (2022)
- [Perceptual Loss with Recognition Model for Single-Channel Enhancement and Robust ASR](https://mlsanity.com/paper/2112.06068/) (2021)
- [Phonetic Feedback for Speech Enhancement With and Without Parallel Speech Data](https://mlsanity.com/paper/2003.01769/) (2020)

## Built on by (most cited first, 2 of 2)

- [Rethinking Voice-Face Correlation: A Geometry View](https://mlsanity.com/paper/2307.13948/) (2023)
- [Psychoacoustic Challenges Of Speech Enhancement On VoIP Platforms](https://mlsanity.com/paper/2310.07161/) (2023)

## Abstract (arXiv)

Despite rapid advancement in recent years, current speech enhancement models often produce speech that differs in perceptual quality from real clean speech. We propose a learning objective that formalizes differences in perceptual quality, by using domain knowledge of acoustic-phonetics. We identify temporal acoustic parameters -- such as spectral tilt, spectral flux, shimmer, etc. -- that are non-differentiable, and we develop a neural network estimator that can accurately predict their time-series values across an utterance. We also model phoneme-specific weights for each feature, as the acoustic parameters are known to show different behavior in different phonemes. We can add this criterion as an auxiliary loss to any model that produces speech, to optimize speech outputs to match the values of clean speech in these features. Experimentally we show that it improves speech enhancement workflows in both time-domain and time-frequency domain, as measured by standard evaluation metrics. We also provide an analysis of phoneme-dependent improvement on acoustic parameters, demonstrating the additional interpretability that our method provides. This analysis can suggest which features are currently the bottleneck for improvement.