# Z-Scores: A Metric for Linguistically Assessing Disfluency Removal

arXiv: https://arxiv.org/abs/2509.20319 · HTML: https://mlsanity.com/paper/2509.20319/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 24 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 11 of 11)

- [Disfluency Detection using a Bidirectional LSTM](https://mlsanity.com/paper/1604.03209/) (2016)
- [Improving Disfluency Detection by Self-Training a Self-Attentive Model](https://mlsanity.com/paper/2004.05323/) (2020)
- [Fluent Translations from Disfluent Speech in End-to-End Speech Translation](https://mlsanity.com/paper/1906.00556/) (2019)
- [End-to-End Speech Recognition and Disfluency Removal](https://mlsanity.com/paper/2009.10298/) (2020)
- [Meta Prompting for AI Systems](https://mlsanity.com/paper/2311.11482/) (2023)
- [Disfluency Detection using a Noisy Channel Model and a Deep Neural Language Model](https://mlsanity.com/paper/1808.09091/) (2018)
- [Neural Constituency Parsing of Speech Transcripts](https://mlsanity.com/paper/1904.08535/) (2019)
- [Combining Self-Training and Self-Supervised Learning for Unsupervised Disfluency Detection](https://mlsanity.com/paper/2010.15360/) (2020)
- [Large Language Models for Dysfluency Detection in Stuttered Speech](https://mlsanity.com/paper/2406.11025/) (2024)
- [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025)
- [Boosting Disfluency Detection with Large Language Model as Disfluency Generator](https://mlsanity.com/paper/2403.08229/) (2024)

## Built on by (most cited first, 1 of 1)

- [The Role of Disfluencies in Speech Translation](https://mlsanity.com/paper/2608.02138/) (2026)

## Abstract (arXiv)

Evaluating disfluency removal in speech requires more than aggregate token-level scores. Traditional word-based metrics such as precision, recall, and F1 (E-Scores) capture overall performance but cannot reveal why models succeed or fail. We introduce Z-Scores, a span-level linguistically-grounded evaluation metric that categorizes system behavior across distinct disfluency types (EDITED, INTJ, PRN). Our deterministic alignment module enables robust mapping between generated text and disfluent transcripts, allowing Z-Scores to expose systematic weaknesses that word-level metrics obscure. By providing category-specific diagnostics, Z-Scores enable researchers to identify model failure modes and design targeted interventions -- such as tailored prompts or data augmentation -- yielding measurable performance improvements. A case study with LLMs shows that Z-Scores uncover challenges with INTJ and PRN disfluencies hidden in aggregate F1, directly informing model refinement strategies.