mlsanityData to 14 Oct 2026

A Language Agnostic Multilingual Streaming On-Device ASR System

arXiv 2208.13916 ↗First posted 29 Aug 2022eess.AS cs.CL cs.SDcited by 9

Not on the map. Why?

Abstract (arXiv)

On-device end-to-end (E2E) models have shown improvements over a conventional model on English Voice Search tasks in both quality and latency. E2E models have also shown promising results for multilingual automatic speech recognition (ASR). In this paper, we extend our previous capacity solution to streaming applications and present a streaming multilingual E2E ASR system that runs fully on device with comparable quality and latency to individual monolingual models. To achieve that, we propose an Encoder Endpointer model and an End-of-Utterance (EOU) Joint Layer for a better quality and latency trade-off. Our system is built in a language agnostic manner allowing it to natively support intersentential code switching in real time. To address the feasibility concerns on large models, we conducted on-device profiling and replaced the time consuming LSTM decoder with the recently developed Embedding decoder. With these changes, we managed to run such a system on a mobile device in less than real time.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 29 Aug 2022.

Builds on

32 of 54 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  4. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  5. 2015Attention-Based Models for Speech Recognitioncited by 401
  6. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  7. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  8. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 234
  9. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 113
  10. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  11. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 101
  12. 2020A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latencycited by 100
20 more
  1. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 97
  2. 2019Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Modelcited by 93
  3. 2019Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modelingcited by 86
  4. 2020FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularizationcited by 69
  5. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 69
  6. 2019Improving RNN Transducer Modeling for End-to-End Speech Recognitioncited by 68
  7. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 63
  8. 2021Scaling End-to-End Models for Large-Scale Multilingual ASRcited by 61
  9. 2019SpecAugment on Large Scale Datasetscited by 60
  10. 2020Towards Fast and Accurate Streaming End-to-End ASRcited by 47
  11. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 44
  12. 2017Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Modelcited by 37
  13. 2021Tied & Reduced RNN-T Decodercited by 36
  14. 2020A New Training Pipeline for an Improved Neural Transducercited by 34
  15. 2018On the End-to-End Solution to Mandarin-English Code-switching Speech Recognitioncited by 30
  16. 2019Massively Multilingual Adversarial Speech Recognitioncited by 28
  17. 2021A Configurable Multilingual Model is All You Need to Recognize All Languagescited by 28
  18. 2020Transformer-Transducers for Code-Switched Speech Recognitioncited by 27
  19. 2018Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languagescited by 26
  20. 2020Less Is More: Improved RNN-T Decoding Using Limited Label Context and Path Mergingcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 1; 2025: 0; 2026: 1.

9 papers.

  1. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASRcited by 17
  2. 2022Scaling Up Deliberation for Multilingual ASRcited by 6
  3. 2022Speech-text based multi-modal training with bidirectional attention for improved speech recognitioncited by 4
  4. 2023Confidence-based Ensembles of End-to-End Speech Recognition Modelscited by 4
  5. 2023SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognitioncited by 4
  6. 2024Efficient Adapter Finetuning for Tail Languages in Streaming Multilingual ASRcited by 4
  7. 2022Random Utterance Concatenation Based Data Augmentation for Improving Short-video Speech Recognitioncited by 3
  8. 2022Internal Language Model Estimation based Adaptive Language Model Fusion for Domain Adaptationcited by 2
  9. 2026The Exact Worst-Case Tail Probability under Bounded Kurtosiscited by 0