# Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency

arXiv: https://arxiv.org/abs/2306.04530 · HTML: https://mlsanity.com/paper/2306.04530/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 28 references that are in the map.
- First posted on arXiv 7 Jun 2023.

## Builds on (most cited first, 5 of 5)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset](https://mlsanity.com/paper/2007.01176/) (2020)
- [WERd: Using Social Text Spelling Variants for Evaluating Dialectal Speech Recognition](https://mlsanity.com/paper/1709.07484/) (2017)

## Built on by (most cited first, 10 of 10)

- [Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning](https://mlsanity.com/paper/2506.17525/) (2025)
- [WER We Stand: Benchmarking Urdu ASR Models](https://mlsanity.com/paper/2409.11252/) (2024)
- [Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages](https://mlsanity.com/paper/2603.00941/) (2026)
- [Robust ASR Error Correction with Conservative Data Filtering](https://mlsanity.com/paper/2407.13300/) (2024)
- [Designing Speech Technologies for Australian Aboriginal English: Opportunities, Risks and Participation](https://mlsanity.com/paper/2503.03186/) (2025)
- [Efficient Adaptation of Multilingual Models for Japanese ASR](https://mlsanity.com/paper/2412.10705/) (2024)
- [Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India](https://mlsanity.com/paper/2604.19151/) (2026)
- [SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation](https://mlsanity.com/paper/2606.02548/) (2026)
- [Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations](https://mlsanity.com/paper/2609.24199/) (2026)
- [Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition](https://mlsanity.com/paper/2609.27289/) (2026)

## Abstract (arXiv)

Word error rate (WER) and character error rate (CER) are standard metrics in Speech Recognition (ASR), but one problem has always been alternative spellings: If one's system transcribes adviser whereas the ground truth has advisor, this will count as an error even though the two spellings really represent the same word. Japanese is notorious for ``lacking orthography'': most words can be spelled in multiple ways, presenting a problem for accurate ASR evaluation. In this paper we propose a new lenient evaluation metric as a more defensible CER measure for Japanese ASR. We create a lattice of plausible respellings of the reference transcription, using a combination of lexical resources, a Japanese text-processing system, and a neural machine translation model for reconstructing kanji from hiragana or katakana. In a manual evaluation, raters rated 95.4% of the proposed spelling variants as plausible. ASR results show that our method, which does not penalize the system for choosing a valid alternate spelling of a word, affords a 2.4%-3.1% absolute reduction in CER depending on the task.