mlsanityData to 7 Oct 2026

LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers

arXiv 2211.02809 ↗First posted 5 Nov 2022cs.CL cs.SD eess.AScited by 10

Not on the map. Why?

Abstract (arXiv)

Automatic speech recognition (ASR) and speech translation (ST) can both use neural transducers as the model structure. It is thus possible to use a single transducer model to perform both tasks. In real-world applications, such joint ASR and ST models may need to be streaming and do not require source language identification (i.e. language-agnostic). In this paper, we propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers. Based on the transducer model structure, we propose four methods, a unified joint and prediction network for multilingual output, a clustered multilingual encoder, target language identification for encoder, and connectionist temporal classification regularization. Experimental results show that LAMASSU not only drastically reduces the model size but also reaches the performances of monolingual ASR and bilingual ST models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 5 Nov 2022.

Builds on

28 of 42 references.

References · 28

  1. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  2. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  3. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  4. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  5. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  6. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  7. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  8. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  9. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  10. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  11. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  12. 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
16 more
  1. 2019Multilingual End-to-End Speech Translationcited by 37
  2. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  3. 2022End-to-End Speech Translation for Code Switched Speechcited by 26
  4. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  5. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  6. 2019Bridging the Gap Between Monaural Speech Enhancement and Recognition with Distortion-Independent Acoustic Modelingcited by 21
  7. 2019Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decodingcited by 21
  8. 2022GigaST: A 10,000-hour Pseudo Speech Translation Corpuscited by 20
  9. 2020Multitask Training with Text Data for End-to-End Speech Recognitioncited by 19
  10. 2020Consistent Transcription and Translation of Speechcited by 14
  11. 2022Code-Switching without Switching: Language Agnostic End-to-End Speech Translationcited by 13
  12. 2021Between Flexibility and Consistency: Joint Generation of Captions and Subtitlescited by 7
  13. 2021FST: the FAIR Speech Translation System for the IWSLT21 Multilingual Shared Taskcited by 6
  14. 2022Joint Generation of Captions and Subtitles with Dual Decodingcited by 5
  15. 2020Speaker Separation Using Speaker Inventories and Estimated Speechcited by 3
  16. 2022Language Tokens: A Frustratingly Simple Approach Improves Zero-Shot Performance of Multilingual Translationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 4; 2025: 2; 2026: 1.

10 papers.

  1. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  2. 2023Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignmentscited by 6
  3. 2023DiariST: Streaming Speech Translation with Speaker Diarizationcited by 5
  4. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
  5. 2025Spatial Speech Translation: Translating Across Space With Binaural Hearablescited by 3
  6. 2023End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translationcited by 2
  7. 2024Streaming Sequence Transduction through Dynamic Compressioncited by 2
  8. 2024Transcribing and Translating, Fast and Slow: Joint Speech Translation and Recognitioncited by 2
  9. 2025HENT-SRT: Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translationcited by 1
  10. 2026Efficient Dialect-Aware Modeling and Conditioning for Low-Resource Taiwanese Hakka Speech Processingcited by 1