# Model as Loss: A Self-Consistent Training Paradigm

arXiv: https://arxiv.org/abs/2505.21156 · HTML: https://mlsanity.com/paper/2505.21156/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 27 May 2025.

## Builds on (most cited first, 18 of 18)

- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [Deep Learning on Image Denoising: An overview](https://mlsanity.com/paper/1912.13171/) (2019)
- [ICASSP 2023 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2303.11510/) (2023)
- [ICASSP 2022 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2202.13288/) (2022)
- [Speech Denoising with Deep Feature Losses](https://mlsanity.com/paper/1806.10522/) (2018)
- [DeepFilterNet2: Towards Real-Time Speech Enhancement on Embedded Devices for Full-Band Audio](https://mlsanity.com/paper/2205.05474/) (2022)
- [SCOREQ: Speech Quality Assessment with Contrastive Regression](https://mlsanity.com/paper/2410.06675/) (2024)
- [SpeechLMScore: Evaluating speech generation using speech language model](https://mlsanity.com/paper/2212.04559/) (2022)
- [A consolidated view of loss functions for supervised deep learning-based speech enhancement](https://mlsanity.com/paper/2009.12286/) (2020)
- [On Loss Functions for Supervised Monaural Time-Domain Speech Enhancement](https://mlsanity.com/paper/1909.01019/) (2019)
- [URGENT Challenge: Universality, Robustness, and Generalizability For Speech Enhancement](https://mlsanity.com/paper/2406.04660/) (2024)
- [ICASSP 2024 Speech Signal Improvement Challenge](https://mlsanity.com/paper/2401.14444/) (2024)
- [Domain Adaptation and Autoencoder Based Unsupervised Speech Enhancement](https://mlsanity.com/paper/2112.05036/) (2021)
- [FINALLY: fast and universal speech enhancement with studio-like quality](https://mlsanity.com/paper/2410.05920/) (2024)
- [Task splitting for DNN-based acoustic echo and noise removal](https://mlsanity.com/paper/2205.06931/) (2022)

## Abstract (arXiv)

Conventional methods for speech enhancement rely on handcrafted loss functions (e.g., time or frequency domain losses) or deep feature losses (e.g., using WavLM or wav2vec), which often fail to capture subtle signal properties essential for optimal performance. To address this, we propose Model as Loss, a novel training paradigm that utilizes the encoder from the same model as a loss function to guide the training. The Model as Loss paradigm leverages the encoder's task-specific feature space, optimizing the decoder to produce output consistent with perceptual and task-relevant characteristics of the clean signal. By using the encoder's learned features as a loss function, this framework enforces self-consistency between the clean reference speech and the enhanced model output. Our approach outperforms pre-trained deep feature losses on standard speech enhancement benchmarks, offering better perceptual quality and robust generalization to both in-domain and out-of-domain datasets.