mlsanityData to 14 Oct 2026

SummaryMixing: A Linear-Complexity Alternative to Self-Attention for Speech Recognition and Understanding

arXiv 2307.07421 ↗First posted 12 Jul 2023cs.CL cs.SD eess.AScited by 11

Not on the map. Why?

Abstract (arXiv)

Modern speech processing systems rely on self-attention. Unfortunately, token mixing with self-attention takes quadratic time in the length of the speech utterance, slowing down inference and training and increasing memory consumption. Cheaper alternatives to self-attention for ASR have been developed, but they fail to consistently reach the same level of accuracy. This paper, therefore, proposes a novel linear-time alternative to self-attention. It summarises an utterance with the mean over vectors for all time steps. This single summary is then combined with time-specific information. We call this method "SummaryMixing". Introducing SummaryMixing in state-of-the-art ASR models makes it feasible to preserve or exceed previous speech recognition performance while making training and inference up to 28% faster and reducing memory use by half.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

32 of 41 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  11. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  12. 2020Efficient Transformers: A Surveycited by 790
20 more
  1. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
  2. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  3. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  4. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  5. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  6. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  7. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  8. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 113
  9. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  10. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 92
  11. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  12. 2021PointMixer: MLP-Mixer for Point Cloud Understandingcited by 71
  13. 2021Fastformer: Additive Attention Can Be All You Needcited by 61
  14. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  15. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 51
  16. 2022WeNet 2.0: More Productive End-to-End Speech Recognition Toolkitcited by 47
  17. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  18. 2020On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformerscited by 24
  19. 2022HyperMixer: An MLP-based Low Cost Alternative to Transformerscited by 13
  20. 2022Locality Matters: A Locality-Biased Linear Attention for Automatic Speech Recognitioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 2.

11 papers.

  1. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  2. 2024Exploring the Capability of Mamba in Speech Applicationscited by 20
  3. 2024Linear-Complexity Self-Supervised Learning for Speech Processingcited by 2
  4. 2024An Analysis of Linear Complexity Attention Substitutes with BEST-RQcited by 2
  5. 2024Linear Time Complexity Conformers with SummaryMixing for Streaming Speech Recognitioncited by 1
  6. 2024Lina-Speech: Gated Linear Attention and Initial-State Tuning for Multi-Sample Prompting Text-To-Speech Synthesiscited by 1
  7. 2025Benchmarking Rotary Position Embeddings for Automatic Speech Recognitioncited by 1
  8. 2025TF-MLPNet: Tiny Real-Time Neural Speech Separationcited by 1
  9. 2025Segmental Attention Decoding With Long Form Acoustic Encodingscited by 0
  10. 2026Windowed SummaryMixing: An Efficient Fine-Tuning of Self-Supervised Learning Models for Low-resource Speech Recognitioncited by 0
  11. 2026Polynomial Mixing for Efficient Self-supervised Speech Encoderscited by 0