# Text Injection for Capitalization and Turn-Taking Prediction in Speech Models

arXiv: https://arxiv.org/abs/2308.07395 · HTML: https://mlsanity.com/paper/2308.07395/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 14 Aug 2023.

## Builds on (most cited first, 26 of 26)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2011.01991/) (2020)
- [MAESTRO: Matched Speech Text Representations through Modality Matching](https://mlsanity.com/paper/2204.03409/) (2022)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020)
- [Joint Speech Recognition and Speaker Diarization via Sequence Transduction](https://mlsanity.com/paper/1907.05337/) (2019)
- [JOIST: A Joint Speech and Text Streaming Model For ASR](https://mlsanity.com/paper/2210.07353/) (2022)
- [Integrating Text Inputs For Training and Adapting RNN Transducer ASR Models](https://mlsanity.com/paper/2202.13155/) (2022)
- [Device-directed Utterance Detection](https://mlsanity.com/paper/1808.02504/) (2018)
- [Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.05354/) (2021)
- [E2E Segmenter: Joint Segmenting and Decoding for Long-Form ASR](https://mlsanity.com/paper/2204.10749/) (2022)
- [Modular Hybrid Autoregressive Transducer](https://mlsanity.com/paper/2210.17049/) (2022)
- [Robust Prediction of Punctuation and Truecasing for Medical ASR](https://mlsanity.com/paper/2007.02025/) (2020)
- [Turn-Taking Prediction for Natural Conversational Speech](https://mlsanity.com/paper/2208.13321/) (2022)
- [Fast and Accurate Capitalization and Punctuation for Automatic Speech Recognition Using Transformer and Chunk Merging](https://mlsanity.com/paper/1908.02404/) (2019)
- [Text-only domain adaptation for end-to-end ASR using integrated text-to-mel-spectrogram generator](https://mlsanity.com/paper/2302.14036/) (2023)
- [Sentence-Select: Large-Scale Language Model Data Selection for Rare-Word Speech Recognition](https://mlsanity.com/paper/2203.05008/) (2022)
- [JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognition](https://mlsanity.com/paper/2302.08583/) (2023)
- [Joint Learning of Correlated Sequence Labelling Tasks Using Bidirectional Recurrent Neural Networks](https://mlsanity.com/paper/1703.04650/) (2017)
- [Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems](https://mlsanity.com/paper/2211.00786/) (2022)
- [Capitalization Normalization for Language Modeling with an Accurate and Efficient Hierarchical RNN Model](https://mlsanity.com/paper/2202.08171/) (2022)
- [Streaming Intended Query Detection using E2E Modeling for Continued Conversation](https://mlsanity.com/paper/2208.13322/) (2022)

## Built on by (most cited first, 1 of 1)

- [Text Injection for Neural Contextual Biasing](https://mlsanity.com/paper/2406.02921/) (2024)

## Abstract (arXiv)

Text injection for automatic speech recognition (ASR), wherein unpaired text-only data is used to supplement paired audio-text data, has shown promising improvements for word error rate. This study examines the use of text injection for auxiliary tasks, which are the non-ASR tasks often performed by an E2E model. In this work, we use joint end-to-end and internal language model training (JEIT) as our text injection algorithm to train an ASR model which performs two auxiliary tasks. The first is capitalization, which is a de-normalization task. The second is turn-taking prediction, which attempts to identify whether a user has completed their conversation turn in a digital assistant interaction. We show results demonstrating that our text injection method boosts capitalization performance for long-tail data, and improves turn-taking detection recall.