mlsanityData to 7 Oct 2026

HyperConformer: Multi-head HyperMixer for Efficient Speech Recognition

arXiv 2305.18281 ↗First posted 29 May 2023cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

State-of-the-art ASR systems have achieved promising results by modeling local and global interactions separately. While the former can be computed efficiently, global interactions are usually modeled via attention mechanisms, which are expensive for long input sequences. Here, we address this by extending HyperMixer, an efficient alternative to attention exhibiting linear complexity, to the Conformer architecture for speech recognition, leading to HyperConformer. In particular, multi-head HyperConformer achieves comparable or higher recognition performance while being more efficient than Conformer in terms of inference speed, memory, parameter count, and available training data. HyperConformer achieves a word error rate of 2.9% on Librispeech test-clean with less than 8M neural parameters and a peak memory during training of 5.7GB, hence trainable with accessible hardware. Encoder speed is between 38% on mid-length speech and 56% on long speech faster than an equivalent Conformer. (The HyperConformer recipe is publicly available in: https://github.com/speechbrain/speechbrain/tree/develop/recipes/LibriSpeech/ASR/transformer/)

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 25 of its 32 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

25 of 32 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Layer Normalizationcited by 5,579
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  12. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
13 more
  1. 2016HyperNetworkscited by 1,003
  2. 2020Efficient Transformers: A Surveycited by 784
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  4. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  5. 2021Pay Attention to MLPscited by 367
  6. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  7. 2021CycleMLP: A MLP-like Architecture for Dense Predictioncited by 126
  8. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  9. 2022E-Branchformer: Branchformer with Enhanced merging for speech recognitioncited by 90
  10. 2022DynaMixer: A Vision MLP Architecture with Dynamic Mixingcited by 28
  11. 2022HyperMixer: An MLP-based Low Cost Alternative to Transformerscited by 13
  12. 2022Federated Self-supervised Speech Representations: Are We There Yet?cited by 10
  13. 2022MLP-ASR: Sequence-length agnostic all-MLP architectures for speech recognitioncited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 1.

5 papers.

  1. 2024Open-Source Conversational AI with SpeechBrain 1.0cited by 55
  2. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representationscited by 19
  3. 2023End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translationcited by 2
  4. 2024An Analysis of Linear Complexity Attention Substitutes with BEST-RQcited by 2
  5. 2025TF-MLPNet: Tiny Real-Time Neural Speech Separationcited by 1