mlsanityData to 7 Oct 2026

H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences

arXiv 2107.11906 ↗First posted 25 Jul 2021cs.LG cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

We describe an efficient hierarchical method to compute attention in the Transformer architecture. The proposed attention mechanism exploits a matrix structure similar to the Hierarchical Matrix (H-Matrix) developed by the numerical analysis community, and has linear run time and memory complexity. We perform extensive experiments to show that the inductive bias embodied by our hierarchical attention is effective in capturing the hierarchical structure in the sequences typical for natural language and vision tasks. Our method is superior to alternative sub-quadratic proposals by over +6 points on average on the Long Range Arena benchmark. It also sets a new SOTA test perplexity on One-Billion Word dataset with 5x fewer model parameters than that of the previous-best Transformer-based models.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 25 Jul 2021.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2017Graph Attention Networkscited by 8,854
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
20 more
  1. 2020Efficient Transformers: A Surveycited by 784
  2. 2018Image Transformercited by 770
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  4. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  5. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  6. 2019Attention Augmented Convolutional Networkscited by 363
  7. 2019Axial Attention in Multidimensional Transformerscited by 350
  8. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  9. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
  10. 2020Sparse Sinkhorn Attentioncited by 250
  11. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  12. 2018Music Transformercited by 207
  13. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 204
  14. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  15. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  16. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  17. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  18. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  19. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  20. 2019Hierarchical Attentional Hybrid Neural Networks for Document Classificationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 10; 2024: 7; 2025: 13; 2026: 5.

47 papers.

Sort:

By area:

  1. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  3. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  4. 2022Memorizing Transformerscited by 219
  5. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  6. 2022Liquid Structural State-Space Modelscited by 134
  7. 2022Block-Recurrent Transformerscited by 107
  8. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  9. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  10. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  11. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  12. 2025Log-Linear Attentioncited by 39
35 more
  1. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  2. 2023Efficient Attention via Control Variatescited by 23
  3. 2021Transformer Uncertainty Estimation with Hierarchical Stochastic Attentioncited by 21
  4. 2025Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systemscited by 21
  5. 2023Sequence Modeling with Multiresolution Convolutional Memorycited by 19
  6. 2023Learning Long Sequences in Spiking Neural Networkscited by 19
  7. 2023Recasting Self-Attention with Holographic Reduced Representationscited by 14
  8. 2022Multi Resolution Analysis (MRA) for Approximate Self-Attentioncited by 11
  9. 2024Wavelet-Based Image Tokenizer for Vision Transformerscited by 10
  10. 2025ZETA: Leveraging Z-order Curves for Efficient Top-k Attentioncited by 9
  11. 2024IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUscited by 8
  12. 2024Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detectioncited by 6
  13. 2025Trainable Log-linear Sparse Attention for Efficient Diffusion Transformerscited by 6
  14. 2021PoNet: Pooling Network for Efficient Token Mixing in Long Sequencescited by 5
  15. 2023Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalabilitycited by 5
  16. 2025Linear Attention with Global Context: A Multipole Attention Mechanism for Vision and Physicscited by 5
  17. 2024LongVQ: Long Sequence Modeling with Vector Quantization on Structured Memorycited by 4
  18. 2025Linear Attention for Efficient Bidirectional Sequence Modelingcited by 4
  19. 2022What Dense Graph Do You Need for Self-Attention?cited by 3
  20. 2023Fast Multipole Attention: A Scalable Multilevel Attention Mechanism for Text and Imagescited by 3
  21. 2024S3Attention: Improving Long Sequence Attention with Smoothed Skeleton Sketchingcited by 2
  22. 2025MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modelingcited by 2
  23. 2025Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretrainingcited by 1
  24. 2025Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problemscited by 1
  25. 2022Global Hierarchical Attention for 3D Point Cloud Analysiscited by 0
  26. 2023A Quantitative Review on Language Model Efficiency Researchcited by 0
  27. 2024Efficient State Space Model via Fast Tensor Convolution and Block Diagonalizationcited by 0
  28. 2025Natively Trainable Sparse Attention for Hierarchical Point Cloud Datasetscited by 0
  29. 2025Customizing the Inductive Biases of Softmax Attention using Structured Matricescited by 0
  30. 2025PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and Memory-Efficient Language Generationcited by 0
  31. 2026Superlinear Multi-Step Attentioncited by 0
  32. 2026Automatic Cardiac Risk Management Classification using large-context Electronic Patients Health Recordscited by 0
  33. 2026BF1: A Causal Dyadic Sparse-Attention Retrofit for Efficient Long-Context Transformerscited by 0
  34. 2026RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitivescited by 0
  35. 2026CEDAR: Error-Bounded Residual Routing for Efficient Long-Context Attentioncited by 0