mlsanityData to 7 Oct 2026

Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments

arXiv 2307.03354 ↗First posted 7 Jul 2023cs.CL cs.SD eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

In real-world applications, users often require both translations and transcriptions of speech to enhance their comprehension, particularly in streaming scenarios where incremental generation is necessary. This paper introduces a streaming Transformer-Transducer that jointly generates automatic speech recognition (ASR) and speech translation (ST) outputs using a single decoder. To produce ASR and ST content effectively with minimal latency, we propose a joint token-level serialized output training method that interleaves source and target words by leveraging an off-the-shelf textual aligner. Experiments in monolingual (it-en) and multilingual (de,es,it-en) settings demonstrate that our approach achieves the best quality-latency balance. With an average ASR latency of 1s and ST latency of 1.3s, our model shows no degradation or even improves output quality compared to separate ASR and ST models, yielding an average improvement of 1.1 WER and 0.4 BLEU in the multilingual case.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 7 Jul 2023.

Builds on

42 of 55 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  5. 2019Green AIcited by 620
  6. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  7. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  8. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  9. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  10. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  11. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  12. 2020SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translationcited by 80
30 more
  1. 2020Streaming automatic speech recognition with the transformer modelcited by 73
  2. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 68
  3. 2020FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularizationcited by 66
  4. 2022Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translationcited by 56
  5. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
  6. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  7. 2018Low-Latency Neural Speech Translationcited by 35
  8. 2021Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASRcited by 31
  9. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  10. 2020Re-translation versus Streaming for Simultaneous Translationcited by 30
  11. 2019Transformer ASR with Contextual Block Processingcited by 28
  12. 2020Consecutive Decoding for Speech-to-text Translationcited by 28
  13. 2020Transformer-Transducers for Code-Switched Speech Recognitioncited by 27
  14. 2019Self-Attention Transducers for End-to-End Speech Recognitioncited by 24
  15. 2019Synchronous Transformers for End-to-End Speech Recognitioncited by 24
  16. 2020Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognitioncited by 23
  17. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  18. 2020Consistent Transcription and Translation of Speechcited by 14
  19. 2022A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capabilitycited by 12
  20. 2022Multi-blank Transducers for Speech Recognitioncited by 11
  21. 2021Streaming Models for Joint Speech Recognition and Translationcited by 10
  22. 2022LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducerscited by 10
  23. 2022Who Are We Talking About? Handling Person Names in Speech Translationcited by 8
  24. 2021Between Flexibility and Consistency: Joint Generation of Captions and Subtitlescited by 7
  25. 2022VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognitioncited by 7
  26. 2018An Operation Sequence Model for Explainable Neural Machine Translationcited by 6
  27. 2021Multi-mode Transformer Transducer with Stochastic Future Contextcited by 6
  28. 2022Joint Generation of Captions and Subtitles with Dual Decodingcited by 5
  29. 2022Defining maximum acceptable latency of AI-enhanced CAI toolscited by 4
  30. 2022Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generationcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 3.

6 papers.

  1. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  2. 2023DiariST: Streaming Speech Translation with Speaker Diarizationcited by 5
  3. 2023Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translationcited by 4
  4. 2023Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approachcited by 2
  5. 2024Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokenscited by 1
  6. 2024Soft Language Identification for Language-Agnostic Many-to-One End-to-End Speech Translationcited by 0