mlsanityData to 7 Oct 2026

Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory

arXiv 2005.08042 ↗First posted 16 May 2020eess.AS cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Transformer-based acoustic modeling has achieved great suc-cess for both hybrid and sequence-to-sequence speech recogni-tion. However, it requires access to the full sequence, and thecomputational cost grows quadratically with respect to the in-put sequence length. These factors limit its adoption for stream-ing applications. In this work, we proposed a novel augmentedmemory self-attention, which attends on a short segment of theinput sequence and a bank of memories. The memory bankstores the embedding information for all the processed seg-ments. On the librispeech benchmark, our proposed methodoutperforms all the existing streamable transformer methods bya large margin and achieved over 15% relative error reduction,compared with the widely used LC-BLSTM baseline. Our find-ings are also confirmed on some large internal datasets.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 10; 2022: 7; 2023: 12; 2024: 6.

35 papers.

Sort:

  1. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  2. 2022Real-time Online Video Detection with Temporal Smoothing Transformerscited by 58
  3. 2024Automatic Speech Recognition using Advanced Deep Learning Approaches: A surveycited by 50
  4. 2023Transformers in Speech Processing: A Surveycited by 38
  5. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  6. 2023Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognitioncited by 20
  7. 2021Dissecting User-Perceived Latency of On-Device E2E Speech Recognitioncited by 18
  8. 2024How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?cited by 17
  9. 2021Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognitioncited by 14
  10. 2021The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021cited by 13
  11. 2023ZeroPrompt: Streaming Acoustic Encoders are Zero-Shot Masked LMscited by 13
  12. 2022CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASRcited by 11
23 more
  1. 2021Wake Word Detection with Streaming Transformerscited by 10
  2. 2021Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latencycited by 9
  3. 2021Collaborative Training of Acoustic Encoders for Speech Recognitioncited by 7
  4. 2023Self-Attention Amortized Distributional Projection Optimization for Sliced Wasserstein Point-Cloud Reconstructioncited by 7
  5. 2024FASST: Fast LLM-based Simultaneous Speech Translationcited by 7
  6. 2021Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognitioncited by 5
  7. 2021Streaming Transformer Transducer Based Speech Recognition Using Non-Causal Convolutioncited by 5
  8. 2023Shiftable Context: Addressing Training-Inference Context Mismatch in Simultaneous Speech Translationcited by 4
  9. 2023Implicit Memory Transformer for Computationally Efficient Simultaneous Speech Translationcited by 4
  10. 2024Overcoming Non-monotonicity in Transducer-based Streaming Generationcited by 4
  11. 2023Improved Training for End-to-End Streaming Automatic Speech Recognition Model with Punctuationcited by 3
  12. 2021Capturing Multi-Resolution Context by Dilated Self-Attentioncited by 2
  13. 2022Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASRcited by 2
  14. 2024Token-Weighted RNN-T for Learning from Flawed Datacited by 2
  15. 2021StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Datacited by 1
  16. 2022SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolutioncited by 1
  17. 2022Pushing the performances of ASR models on English and Spanish accentscited by 1
  18. 2024Transducer Consistency Regularization for Speech to Text Applicationscited by 1
  19. 2022Dynamic Latency for CTC-Based Streaming Automatic Speech Recognition With Emformercited by 0
  20. 2022FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognitioncited by 0
  21. 2023A low latency attention module for streaming self-supervised speech representation learningcited by 0
  22. 2023DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer ASRcited by 0
  23. 2023Memory-augmented conformer for improved end-to-end long-form ASRcited by 0