mlsanityData to 14 Oct 2026

Hierarchical Transformers Are More Efficient Language Models

arXiv 2110.13711 ↗First posted 26 Oct 2021cs.LG cs.CLcited by 102

Not on the map. Why?

Abstract (arXiv)

Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers can handle long sequences which allows them to produce long coherent outputs: full paragraphs produced by GPT-3 or well-structured images produced by DALL-E. These large language models are impressive but also very inefficient and costly, which limits their applications and accessibility. We postulate that having an explicit hierarchical architecture is the key to Transformers that efficiently handle long sequences. To verify this claim, we first study different ways to downsample and upsample activations in Transformers so as to make them hierarchical. We use the best performing upsampling and downsampling layers to create Hourglass - a hierarchical Transformer language model. Hourglass improves upon the Transformer baseline given the same amount of computation and can yield the same results as Transformers more efficiently. In particular, Hourglass sets new state-of-the-art for Transformer models on the ImageNet32 generation task and improves language modeling efficiency on the widely studied enwik8 benchmark.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

33 of 40 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  12. 2020Reformer: The Efficient Transformercited by 1,668
21 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2021Variational Diffusion Modelscited by 1,153
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  5. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  6. 2016Pixel Recurrent Neural Networkscited by 816
  7. 2018Image Transformercited by 772
  8. 2020Jukebox: A Generative Model for Musiccited by 542
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  10. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  11. 2019Axial Attention in Multidimensional Transformerscited by 358
  12. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  13. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  14. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  15. 2019Adaptive Attention Span in Transformerscited by 149
  16. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  17. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  18. 2021Densely connected normalizing flowscited by 49
  19. 2019BP-Transformer: Modelling Long-Range Context via Binary Partitioningcited by 38
  20. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  21. 2020Multi-scale Transformer Language Modelscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 11; 2023: 10; 2024: 17; 2025: 33; 2026: 31.

102 papers.

Sort:

By area:

  1. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  2. 2024Scalable High-Resolution Pixel-Space Image Synthesis with Hourglass Diffusion Transformerscited by 118
  3. 2024Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scalecited by 89
  4. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  5. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  6. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  7. 2025DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learningcited by 65
  8. 2022General-purpose, long-context autoregressive modeling with Perceiver ARcited by 60
  9. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  10. 2022Unified Fully and Timestamp Supervised Temporal Action Segmentation via Sequence to Sequence Translationcited by 55
  11. 2022Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecastingcited by 54
  12. 2022Temporally Consistent Transformers for Video Generationcited by 44
38 more
  1. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  2. 2022G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Modelcited by 38
  3. 2025ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understandingcited by 38
  4. 2023How Much Temporal Long-Term Context is Needed for Action Segmentation?cited by 36
  5. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 34
  6. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 33
  7. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  8. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  9. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 28
  10. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  11. 2025Mesh-RFT: Enhancing Mesh Generation via Fine-grained Reinforcement Fine-Tuningcited by 22
  12. 2022HiP: Hierarchical Perceivercited by 15
  13. 2025From Bytes to Ideas: Language Modeling with Autoregressive U-Netscited by 15
  14. 2025iFlame: Interleaving Full and Linear Attention for Efficient Mesh Generationcited by 13
  15. 2025QuadGPT: Native Quadrilateral Mesh Generation with Autoregressive Modelscited by 12
  16. 2025The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Modelscited by 11
  17. 2026Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Modelscited by 11
  18. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 9
  19. 2023Toucan: Token-Aware Character Level Language Modelingcited by 8
  20. 2025VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Modelscited by 8
  21. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  22. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 8
  23. 2024Beyond Next Token Prediction: Patch-Level Training for Large Language Modelscited by 7
  24. 2025Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modelingcited by 7
  25. 2022Generating Coherent Narratives by Learning Dynamic and Discrete Entity States with a Contrastive Frameworkcited by 6
  26. 2026LATO.2: Factorized 3D Mesh Generation with Vertex and Topology Flowcited by 6
  27. 2023Recursion in Recursion: Two-Level Nested Recursion for Length Generalization with Scalabilitycited by 5
  28. 2025Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformerscited by 5
  29. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 5
  30. 2025v-PuNNs: van der Put Neural Networks for Transparent Ultrametric Representation Learningcited by 5
  31. 2023From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understandingcited by 4
  32. 2024CryoFM: A Flow-based Foundation Model for Cryo-EM Densitiescited by 4
  33. 2023Hierarchical Attention Encoder Decodercited by 3
  34. 2024Efficient Hierarchical Transformers for Representing Log Datacited by 3
  35. 2025Controllably Efficient Language Modelscited by 3
  36. 2026Proxy Compression for Language Modelingcited by 3
  37. 2026Mesh-Pro: Asynchronous Advantage-guided Ranking Preference Optimization for Artist-style Quadrilateral Mesh Generationcited by 3
  38. 2026PixARMesh: Autoregressive Mesh-Native Single-View Scene Reconstructioncited by 3

50 most cited of 102 All 102 (JSON)