mlsanityData to 7 Oct 2026

Best-of-N TTS Evaluation is Confounded by ASR Family Alignment

arXiv 2607.08256 ↗First posted 9 Jul 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Best-of-N (BoN) inference improves content consistency in zero-shot text-to-speech by selecting among multiple candidates with an automatic speech recognition (ASR) verifier. We identify an evaluation confound: the apparent quality of a verifier depends strongly on the ASR family used for evaluation. On LibriSpeech-PC with F5-TTS, verifier rankings vary substantially across Whisper, wav2vec 2.0, and HuBERT evaluators, while same-family verifier and evaluator pairs recover considerably more oracle headroom than cross-family pairs despite highly similar representations. This pattern suggests identity- or lineage-level coupling rather than general representational similarity. To mitigate this bias, we propose two cross-family rank ensembles: rank averaging and conjunctive max-rank. Both improve mean word error rate across independent evaluators without degrading automatic similarity or quality metrics, and the best ensemble achieves a 12% relative WER reduction over F5-TTS at N=10. These findings motivate cross-evaluator triangulation as a more reliable default for reporting BoN TTS performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 27 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

23 of 27 references.

References · 23

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  8. 2022UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022cited by 381
  9. 2024F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matchingcited by 353
  10. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 293
  11. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 282
  12. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 222
11 more
  1. 2021NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasetscited by 189
  2. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  3. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 149
  4. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  5. 2024The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speechcited by 78
  6. 2021Generalization Ability of MOS Prediction Networkscited by 69
  7. 2023LibriSpeech-PC: Benchmark for Evaluation of Punctuation and Capitalization Capabilities of end-to-end ASR Modelscited by 46
  8. 2024SpeechAlign: Aligning Speech Generation to Human Preferencescited by 45
  9. 2024Robust Zero-Shot Text-to-Speech Synthesis with Reverse Inference Optimizationcited by 10
  10. 2025Scaling Flaws of Verifier-Guided Search in Mathematical Reasoningcited by 10
  11. 2025Pseudo-Autoregressive Neural Codec Language Models for Efficient Zero-Shot Text-to-Speech Synthesiscited by 10

Built on by

Not cited yet.