mlsanityData to 7 Oct 2026

A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability

arXiv 2211.02499 ↗First posted 4 Nov 2022cs.CL cs.AI cs.SDcited by 12

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce our work of building a Streaming Multilingual Speech Model (SM2), which can transcribe or translate multiple spoken languages into texts of the target language. The backbone of SM2 is Transformer Transducer, which has high streaming capability. Instead of human labeled speech translation (ST) data, SM2 models are trained using weakly supervised data generated by converting the transcriptions in speech recognition corpora with a machine translation service. With 351 thousand hours of anonymized speech training data from 25 languages, SM2 models achieve comparable or even better ST quality than some recent popular large-scale non-streaming speech models. More importantly, we show that SM2 has the truly zero-shot capability when expanding to new target languages, yielding high quality ST results for {source-speech, target-text} pairs that are not seen during training.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 25 of its 46 references that are in the map. First posted on arXiv 4 Nov 2022.

Builds on

25 of 46 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2019Green AIcited by 620
  4. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  5. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  6. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  7. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  8. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  9. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 103
  10. 2020SimulEval: An Evaluation Toolkit for Simultaneous Translationcited by 94
  11. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  12. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 92
13 more
  1. 2019Monotonic Multihead Attentioncited by 91
  2. 2017Multilingual Speech Recognition With A Single End-To-End Modelcited by 87
  3. 2017Monotonic Chunkwise Attentioncited by 82
  4. 2020Speech Translation and the End-to-End Promise: Taking Stock of Where We Arecited by 77
  5. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  6. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  7. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  8. 2019Multilingual End-to-End Speech Translationcited by 37
  9. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  10. 2020Streaming Simultaneous Speech Translation with Augmented Memory Transformercited by 30
  11. 2021A Configurable Multilingual Model is All You Need to Recognize All Languagescited by 28
  12. 2021Scaling ASR Improves Zero and Few Shot Learningcited by 15
  13. 2020High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Modelcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 6; 2024: 4; 2025: 2.

12 papers.

  1. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  2. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  3. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 24
  4. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  5. 2023ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translationcited by 9
  6. 2023Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignmentscited by 6
  7. 2023DiariST: Streaming Speech Translation with Speaker Diarizationcited by 5
  8. 2023Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approachcited by 2
  9. 2024Target word activity detector: An approach to obtain ASR word boundaries without lexiconcited by 1
  10. 2024CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translationcited by 1
  11. 2025PHRASED: Phrase Dictionary Biasing for Speech Translationcited by 1
  12. 2024Soft Language Identification for Language-Agnostic Many-to-One End-to-End Speech Translationcited by 0