mlsanityData to 7 Oct 2026

JOIST: A Joint Speech and Text Streaming Model For ASR

arXiv 2210.07353 ↗First posted 13 Oct 2022cs.CL cs.SD eess.AScited by 24

Not on the map. Why?

Abstract (arXiv)

We present JOIST, an algorithm to train a streaming, cascaded, encoder end-to-end (E2E) model with both speech-text paired inputs, and text-only unpaired inputs. Unlike previous works, we explore joint training with both modalities, rather than pre-training and fine-tuning. In addition, we explore JOIST using a streaming E2E model with an order of magnitude more data, which are also novelties compared to previous works. Through a series of ablation studies, we explore different types of text modeling, including how to model the length of the text sequence and the appropriate text sub-word unit representation. We find that best text representation for JOIST improves WER across a variety of search and rare-word test sets by 4-14% relative, compared to a model not trained with text. In addition, we quantitatively show that JOIST maintains streaming capabilities, which is important for good user-level experience.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 46 of its 69 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

46 of 69 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  9. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  11. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  12. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
34 more
  1. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  2. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  3. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  4. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  5. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  6. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  7. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  8. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  9. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  10. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  11. 2019Recognizing long-form speech using streaming end-to-end modelscited by 77
  12. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  13. 2017Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Modelscited by 68
  14. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  15. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
  16. 2020FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularizationcited by 66
  17. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  18. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  19. 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 51
  20. 2017Listening while Speaking: Speech Chain by Deep Learningcited by 46
  21. 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
  22. 2021Tied & Reduced RNN-T Decodercited by 35
  23. 2020A New Training Pipeline for an Improved Neural Transducercited by 34
  24. 2020Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognitioncited by 34
  25. 2018Cycle-consistency training for end-to-end speech recognitioncited by 31
  26. 2022Pseudo Label Is Better Than Human Labelcited by 25
  27. 2020Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpuscited by 24
  28. 2021Injecting Text in Self-Supervised Speech Pretrainingcited by 24
  29. 2019Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognitioncited by 23
  30. 2020Joint Masked CPC and CTC Training for ASRcited by 20
  31. 2022Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systemscited by 20
  32. 2022Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizerscited by 19
  33. 2013Feature Learning in Deep Neural Networks - Studies on Speech Recognition Taskscited by 14
  34. 2022USTED: Improving ASR with a Unified Speech and Text Encoder-Decodercited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 14; 2024: 6; 2025: 1; 2026: 2.

24 papers.

Sort:

By area:

  1. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  2. 2023End-to-End Speech Recognition: A Surveycited by 127
  3. 2023Efficient Domain Adaptation for Speech Foundation Modelscited by 13
  4. 2023Massively Multilingual Shallow Fusion with Large Language Modelscited by 10
  5. 2023JEIT: Joint End-to-End Model and Internal Language Model Training for Speech Recognitioncited by 9
  6. 2022Mu2SLAM: Multitask, Multilingual Speech and Language Modelscited by 7
  7. 2023Understanding Shared Speech-Text Representationscited by 6
  8. 2023Improving Large-scale Deep Biasing with Phoneme Features and Text-only Data in Streaming Transducercited by 6
  9. 2023Text-only Domain Adaptation using Unified Speech-Text Representation in Transducercited by 5
  10. 2023Improving Joint Speech-Text Representations Without Alignmentcited by 4
  11. 2024Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuningcited by 4
  12. 2023Using Text Injection to Improve Recognition of Personal Identifiers in Speechcited by 3
12 more
  1. 2024Text Injection for Neural Contextual Biasingcited by 3
  2. 2023Augmenting text for spoken language understanding with Large Language Modelscited by 2
  3. 2024ASTRA: Aligning Speech and Text Representations for Asr without Samplingcited by 2
  4. 2024An efficient text augmentation approach for contextualized Mandarin speech recognitioncited by 2
  5. 2024CJST: CTC Compressor based Joint Speech and Text Training for Decoder-Only ASRcited by 2
  6. 2023A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scalecited by 1
  7. 2023Text Injection for Capitalization and Turn-Taking Prediction in Speech Modelscited by 1
  8. 2025A Non-autoregressive Model for Joint STT and TTScited by 1
  9. 2023Text-Only Domain Adaptation for End-to-End Speech Recognition through Down-Sampling Acoustic Representationcited by 0
  10. 2024High-precision Voice Search Query Correction via Retrievable Speech-text Embedingscited by 0
  11. 2026Text-Utilization for Encoder-dominated Speech Recognition Modelscited by 0
  12. 2026Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleavingcited by 0