# How do Hyenas deal with Human Speech? Speech Recognition and Translation with ConfHyena

arXiv: https://arxiv.org/abs/2402.13208 · HTML: https://mlsanity.com/paper/2402.13208/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [End-to-End Automatic Speech Translation of Audiobooks](https://mlsanity.com/paper/1802.04200/) (2018)
- [Bridging the Modality Gap for Speech-to-Text Translation](https://mlsanity.com/paper/2010.14920/) (2020)
- [Encoding word order in complex embeddings](https://mlsanity.com/paper/1912.12333/) (2019)
- [CTC-based Compression for Direct Speech Translation](https://mlsanity.com/paper/2102.01578/) (2021)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Unraveling the Hidden Environmental Impacts of AI Solutions for Environment](https://mlsanity.com/paper/2110.11822/) (2021)
- [Adaptive Feature Selection for End-to-End Speech Translation](https://mlsanity.com/paper/2010.08518/) (2020)
- [Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation](https://mlsanity.com/paper/1906.01199/) (2019)
- [Speechformer: Reducing Information Loss in Direct Speech Translation](https://mlsanity.com/paper/2109.04574/) (2021)
- [When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP](https://mlsanity.com/paper/2303.16166/) (2023)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [Efficiency Pentathlon: A Standardized Arena for Efficiency Evaluation](https://mlsanity.com/paper/2307.09701/) (2023)
- [Efficient Transformer for Direct Speech Translation](https://mlsanity.com/paper/2107.03069/) (2021)

## Built on by (most cited first, 1 of 1)

- [Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond](https://mlsanity.com/paper/2410.12982/) (2024)

## Abstract (arXiv)

The attention mechanism, a cornerstone of state-of-the-art neural models, faces computational hurdles in processing long sequences due to its quadratic complexity. Consequently, research efforts in the last few years focused on finding more efficient alternatives. Among them, Hyena (Poli et al., 2023) stands out for achieving competitive results in both language modeling and image classification, while offering sub-quadratic memory and computational complexity. Building on these promising results, we propose ConfHyena, a Conformer whose encoder self-attentions are replaced with an adaptation of Hyena for speech processing, where the long input sequences cause high computational costs. Through experiments in automatic speech recognition (for English) and translation (from English into 8 target languages), we show that our best ConfHyena model significantly reduces the training time by 27%, at the cost of minimal quality degradation (~1%), which, in most cases, is not statistically significant.