mlsanityData to 7 Oct 2026

Hyena Hierarchy: Towards Larger Convolutional Language Models

arXiv 2302.10866 ↗First posted 21 Feb 2023cs.LG cs.CLcited by 388

Not on the map. Why?

Cited by 61 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the core building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.

Key facts. Cited by 388 papers in the map (data through 7 Oct 2026). Builds on 47 of its 66 references that are in the map. First posted on arXiv 21 Feb 2023.

Builds on

47 of 66 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  12. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
35 more
  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  6. 2020Reformer: The Efficient Transformercited by 1,658
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2017Random Erasing Data Augmentationcited by 1,235
  10. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  11. 2016Deep Networks with Stochastic Depthcited by 973
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  13. 2022In-context Learning and Induction Headscited by 930
  14. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  15. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  16. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  17. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  18. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  19. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  20. 2022MaxViT: Multi-Axis Vision Transformercited by 476
  21. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  22. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  23. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  24. 2022Long Range Language Modeling via Gated State Spacescited by 265
  25. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  26. 2020Dissecting Neural ODEscited by 150
  27. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  28. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  29. 2021An Attention Free Transformercited by 111
  30. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  31. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  32. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  33. 2019Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizationscited by 73
  34. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  35. 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spacescited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 63; 2024: 149; 2025: 93; 2026: 83.

388 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  5. 2024Large Language Models: A Surveycited by 506
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  7. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  8. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  9. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  10. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  11. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  12. 2023Challenges and Applications of Large Language Modelscited by 251
38 more
  1. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  2. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  3. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  6. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  9. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  10. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  11. 2023Protein Design with Guided Discrete Diffusioncited by 154
  12. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  13. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  14. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  15. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  16. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  17. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  18. 2024Understanding Emergent Abilities of Language Models from the Loss Perspectivecited by 94
  19. 2024The Hidden Attention of Mamba Modelscited by 88
  20. 2024Advancing Multimodal Medical Capabilities of Geminicited by 88
  21. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  22. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  23. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Datacited by 84
  24. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time seriescited by 81
  25. 2023Diffusion Models Without Attentioncited by 75
  26. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  27. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  28. 2023RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmarkcited by 70
  29. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  30. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  31. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  32. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  33. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  34. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
  35. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  36. 2024Rewrite the Starscited by 55
  37. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 55
  38. 2024Linearizing Large Language Modelscited by 51

50 most cited of 388 All 388 (JSON)