mlsanityData to 14 Oct 2026

Linear Time Complexity Conformers with SummaryMixing for Streaming Speech Recognition

arXiv 2409.07165 ↗First posted 11 Sep 2024cs.SD cs.AI eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

Automatic speech recognition (ASR) with an encoder equipped with self-attention, whether streaming or non-streaming, takes quadratic time in the length of the speech utterance. This slows down training and decoding, increase their cost, and limit the deployment of the ASR in constrained devices. SummaryMixing is a promising linear-time complexity alternative to self-attention for non-streaming speech recognition that, for the first time, preserves or outperforms the accuracy of self-attention models. Unfortunately, the original definition of SummaryMixing is not suited to streaming speech recognition. Hence, this work extends SummaryMixing to a Conformer Transducer that works in both a streaming and an offline mode. It shows that this new linear-time complexity speech encoder outperforms self-attention in both scenarios while requiring less compute and memory during training and decoding.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

22 of 29 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  6. 2020Efficient Transformers: A Surveycited by 790
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  8. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  9. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  10. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 125
  11. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 113
  12. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
10 more
  1. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  2. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 80
  3. 2020A Better and Faster End-to-End Model for Streaming ASRcited by 69
  4. 2021Fastformer: Additive Attention Can Be All You Needcited by 61
  5. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  6. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  7. 2020On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformerscited by 24
  8. 2023SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understandingcited by 11
  9. 2023Dynamic Chunk Convolution for Unified Streaming and Non-Streaming Conformer ASRcited by 7
  10. 2024Linear-Complexity Self-Supervised Learning for Speech Processingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Do we really need Self-Attention for Streaming Automatic Speech Recognition?cited by 0