mlsanityData to 14 Oct 2026

Blockwise Self-Attention for Long Document Understanding

arXiv 1911.02972 ↗First posted 7 Nov 2019cs.CL cs.LGcited by 159

Not on the map. Why?

Abstract (arXiv)

We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.

Key facts. Cited by 159 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 33; 2022: 38; 2023: 33; 2024: 40; 2025: 9; 2026: 6.

159 papers.

Sort:

By area:

  1. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  2. 2021Random Feature Attentioncited by 354
  3. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  4. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  5. 2021XCiT: Cross-Covariance Image Transformerscited by 338
  6. 2022Long Range Language Modeling via Gated State Spacescited by 265
  7. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  8. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  9. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  10. 2023Efficient Large Language Models: A Surveycited by 186
  11. 2022Separable Self-attention for Mobile Vision Transformerscited by 163
  12. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
38 more
  1. 2021Self-attention Does Not Need O(n2) Memorycited by 154
  2. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  3. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 116
  4. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  5. 2021Finetuning Pretrained Transformers into RNNscited by 96
  6. 2021Luna: Linear Unified Nested Attentioncited by 96
  7. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  8. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  9. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  10. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  11. 2021Token Pooling in Vision Transformerscited by 69
  12. 2025Training-free and Adaptive Sparse Attention for Efficient Long Video Generationcited by 69
  13. 2022Single Stage Virtual Try-on via Deformable Attention Flowscited by 65
  14. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  15. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 63
  16. 2022ClusterGNN: Cluster-based Coarse-to-Fine Graph Neural Network for Efficient Feature Matchingcited by 62
  17. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  18. 2021Memory-efficient Transformers via Top-k Attentioncited by 60
  19. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 52
  20. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 51
  21. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 49
  22. 2021Efficient Visual Tracking with Exemplar Transformerscited by 48
  23. 2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47
  24. 2023A Survey on Long Text Modeling with Transformerscited by 47
  25. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  26. 2024Is Mamba Capable of In-Context Learning?cited by 45
  27. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  28. 2023On Sparse Modern Hopfield Modelcited by 42
  29. 2022ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Designcited by 41
  30. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  31. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  32. 2024Foundation Models for Music: A Surveycited by 37
  33. 2023How Much Temporal Long-Term Context is Needed for Action Segmentation?cited by 36
  34. 2021Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modelingcited by 35
  35. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 33
  36. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  37. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  38. 2023OcTr: Octree-based Transformer for 3D Object Detectioncited by 32

50 most cited of 159 All 159 (JSON)