# Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition

arXiv: https://arxiv.org/abs/2609.27289 · HTML: https://mlsanity.com/paper/2609.27289/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023)
- [JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis](https://mlsanity.com/paper/1711.00354/) (2017)
- [Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency](https://mlsanity.com/paper/2306.04530/) (2023)
- [Construction of a Large-scale Japanese ASR Corpus on TV Recordings](https://mlsanity.com/paper/2103.14736/) (2021)
- [Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition](https://mlsanity.com/paper/2409.00815/) (2024)
- [Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding](https://mlsanity.com/paper/2412.16507/) (2024)
- [A Comparative Study on Neural Architectures and Training Methods for Japanese Speech Recognition](https://mlsanity.com/paper/2106.05111/) (2021)
- [Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimation](https://mlsanity.com/paper/2305.17846/) (2023)
- [Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognition](https://mlsanity.com/paper/2509.04488/) (2025)
- [Benchmarking Large Language Models for Grapheme-to-Phoneme Conversion: A Japanese Case Study](https://mlsanity.com/paper/2606.22009/) (2026)
- [Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis](https://mlsanity.com/paper/2606.25369/) (2026)
- [A Dual-Decoder Conformer for Multilingual Speech Recognition](https://mlsanity.com/paper/2109.03277/) (2021)
- [Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotation](https://mlsanity.com/paper/2506.07646/) (2025)
- [Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing](https://mlsanity.com/paper/2603.10587/) (2026)
- [Beyond Acoustic Prefixes: Persistent Grounding in Serialized Acoustic Memory for LLM-Based Multi-Talker Speech Recognition](https://mlsanity.com/paper/2603.27205/) (2026)
- [YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese](https://mlsanity.com/paper/2607.00664/) (2026)

## Abstract (arXiv)

Conventional Japanese automatic speech recognition (ASR) is supervised by an orthographic transcript, although the same written form can correspond to different lexical readings realized in speech. Such utterances receive an identical target, so their reading distinction is absent from the supervision interface and cannot be recovered reliably by post-hoc text-only grapheme-to-phoneme conversion. We present Ruby-ASR, which refines the conventional target into a span-bound orthographic--lexical-reading sequence. Unlike separate full-sentence orthographic and phonological outputs, the ruby representation locally binds each written span to its realized reading and permits deterministic recovery of both views. We instantiate the target under subtitle-style and verbatim-style transcription conventions using a Qwen3-ASR backbone; a mora-level CTC objective provides auxiliary monotonic reading supervision. The experimental results across five Japanese benchmarks show that refining the recognition target can improve lexical-reading recovery without sacrificing readable orthographic transcription. We release the checkpoints and inference code.