# PashtoTTS-Bench: automated screening for low-resource non-Latin-script text-to-speech

arXiv: https://arxiv.org/abs/2605.26978 · HTML: https://mlsanity.com/paper/2605.26978/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 26 May 2026.

## Builds on (most cited first, 10 of 10)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022](https://mlsanity.com/paper/2204.02152/) (2022)
- [VoxLingua107: a Dataset for Spoken Language Recognition](https://mlsanity.com/paper/2011.12998/) (2020)
- [The Zero Resource Speech Challenge 2021: Spoken language modelling](https://mlsanity.com/paper/2104.14700/) (2021)
- [The Zero Resource Speech Challenge 2017](https://mlsanity.com/paper/1712.04313/) (2017)
- [Benchmarking Multilingual Speech Models on Pashto: Zero-Shot ASR, Script Failure, and Cross-Domain Evaluation](https://mlsanity.com/paper/2604.04598/) (2026)
- [Script collapse in multilingual ASR: A reference-free metric and 100-pair benchmark](https://mlsanity.com/paper/2604.08786/) (2026)

## Built on by (most cited first, 1 of 1)

- [Domain-Specific Evaluation of Text-to-Speech Systems: A Multi-Metric Benchmarking Study](https://mlsanity.com/paper/2608.02235/) (2026)

## Abstract (arXiv)

Text-to-speech (TTS) evaluation for low-resource non-Latin-script languages can fail when it relies on a single ASR round-trip word error rate (WER). A system may produce no audio, speak a neighbouring language, preserve target script text only in an ASR transcript, or sound unnatural to native listeners. We introduce INSV (Intelligibility, Naturalness, Script fidelity, and Verification), a reporting framework that separates these cases. This paper reports INSV-A, the automated screening subset: synthesis completion, ASR WER/CER, transcript Script Fidelity Rate, and audio language identification. Native MOS and phonetic annotation are specified but not claimed in this release. We instantiate INSV-A as PashtoTTS-Bench, a dated benchmark for Pashto TTS. The April-May 2026 run evaluates Edge GulNawaz, Edge Latifa, OmniVoice clone, OmniVoice auto, and an Urdu negative control on 200 FLEURS and 200 filtered Common Voice 24 prompts. Under the independent omniASR_CTC_300M_v2, OmniVoice auto has the lowest WER (24.1% FLEURS, 27.4% CV24), followed by Edge GulNawaz (32.8%, 39.5%), Edge Latifa (35.6%, 47.7%), and OmniVoice clone (45.4%, 34.8%). WER below the natural-speech baseline reflects clean synthetic audio and should not be read as better than native speech. Whisper Large V3 returns 0.0% Pashto labels on checked Pashto TTS audio, while MMS-LID-4017 and SpeechBrain VoxLingua107 separate Pashto outputs from the Urdu control. The release provides provider metadata, per-sentence scores, LID audits, failure logs, and scripts for adding systems.