# Multi-Metric Preference Alignment for Generative Speech Restoration

arXiv: https://arxiv.org/abs/2508.17229 · HTML: https://mlsanity.com/paper/2508.17229/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 41 of its 53 references that are in the map.
- First posted on arXiv 24 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Preference alignment for recommenders and RAG](https://mlsanity.com/alignment/preference-alignment-for-recommenders-and-rag/)

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Improving Video Generation with Human Feedback](https://mlsanity.com/paper/2501.13918/) (2025)
- [Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation](https://mlsanity.com/paper/2407.05361/) (2024)
- [NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets](https://mlsanity.com/paper/2104.09494/) (2021)
- [DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors](https://mlsanity.com/paper/2110.01763/) (2021)
- [MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer](https://mlsanity.com/paper/2409.00750/) (2024)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results](https://mlsanity.com/paper/2005.13981/) (2020)
- [MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement](https://mlsanity.com/paper/1905.04874/) (2019)
- [Speech Enhancement with Score-Based Generative Models in the Complex STFT Domain](https://mlsanity.com/paper/2203.17004/) (2022)
- [SpeechX: Neural Codec Language Model as a Versatile Speech Transformer](https://mlsanity.com/paper/2308.06873/) (2023)
- [Generative Pre-training for Speech with Flow Matching](https://mlsanity.com/paper/2310.16338/) (2023)
- [SpeechAlign: Aligning Speech Generation to Human Preferences](https://mlsanity.com/paper/2404.05600/) (2024)
- [AudioSR: Versatile Audio Super-resolution at Scale](https://mlsanity.com/paper/2309.07314/) (2023)
- [SpeechBERTScore: Reference-Aware Automatic Evaluation of Speech Generation Leveraging NLP Evaluation Metrics](https://mlsanity.com/paper/2401.16812/) (2024)
- [VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration](https://mlsanity.com/paper/2204.05841/) (2022)
- [Analysing Diffusion-based Generative Approaches versus Discriminative Approaches for Speech Restoration](https://mlsanity.com/paper/2211.02397/) (2022)
- [MusicRL: Aligning Music Generation to Human Preferences](https://mlsanity.com/paper/2402.04229/) (2024)
- [Amphion: An Open-Source Audio, Music and Speech Generation Toolkit](https://mlsanity.com/paper/2312.09911/) (2023)
- [Koel-TTS: Enhancing LLM based Speech Generation with Preference Alignment and Classifier Free Guidance](https://mlsanity.com/paper/2502.05236/) (2025)
- [LeVo: High-Quality Song Generation with Multi-Preference Alignment](https://mlsanity.com/paper/2506.07520/) (2025)
- [MaskSR: Masked Language Model for Full-band Speech Restoration](https://mlsanity.com/paper/2406.02092/) (2024)
- [F5R-TTS: Improving Flow-Matching based Text-to-Speech with Group Relative Policy Optimization](https://mlsanity.com/paper/2504.02407/) (2025)
- [AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement](https://mlsanity.com/paper/2501.15417/) (2025)
- [Preference Alignment Improves Language Model-Based TTS](https://mlsanity.com/paper/2409.12403/) (2024)
- [GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling](https://mlsanity.com/paper/2502.02942/) (2025)

## Built on by (most cited first, 7 of 7)

- [FlowSE-GRPO: Training Flow Matching Speech Enhancement via Online Reinforcement Learning](https://mlsanity.com/paper/2601.16483/) (2026)
- [Schr"odinger Bridge Mamba for One-Step Speech Enhancement](https://mlsanity.com/paper/2510.16834/) (2025)
- [Investigating training objective for flow matching-based speech enhancement](https://mlsanity.com/paper/2512.10382/) (2025)
- [GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model](https://mlsanity.com/paper/2512.20978/) (2025)
- [Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models](https://mlsanity.com/paper/2605.27383/) (2026)
- [Reliable Neural-Codec Text-to-Speech by ASR Self-Verification and Distillation: Near-Zero Catastrophic Failures Across Models and Codecs](https://mlsanity.com/paper/2606.18323/) (2026)
- [Transcript-Supervised Post-Training of Generative Speech Enhancement on Real Recordings via Reinforce Adjoint Matching](https://mlsanity.com/paper/2609.29405/) (2026)

## Abstract (arXiv)

Recent generative models have significantly advanced speech restoration tasks, yet their training objectives often misalign with human perceptual preferences, resulting in suboptimal quality. While post-training alignment has proven effective in other generative domains like text and image generation, its application to generative speech restoration remains largely under-explored. This work investigates the challenges of applying preference-based post-training to this task, focusing on how to define a robust preference signal and curate high-quality data to avoid reward hacking. To address these challenges, we propose a multi-metric preference alignment strategy. We construct a new dataset, GenSR-Pref, comprising 80K preference pairs, where each chosen sample is unanimously favored by a complementary suite of metrics covering perceptual quality, signal fidelity, content consistency, and timbre preservation. This principled approach ensures a holistic preference signal. Applying Direct Preference Optimization (DPO) with our dataset, we observe consistent and significant performance gains across three diverse generative paradigms: autoregressive models (AR), masked generative models (MGM), and flow-matching models (FM) on various restoration benchmarks, in both objective and subjective evaluations. Ablation studies confirm the superiority of our multi-metric strategy over single-metric approaches in mitigating reward hacking. Furthermore, we demonstrate that our aligned models can serve as powerful ''data annotators'', generating high-quality pseudo-labels to serve as a supervision signal for traditional discriminative models in data-scarce scenarios like singing voice restoration. Demo Page:https://gensr-pref.github.io