mlsanityData to 14 Oct 2026

BP-Transformer: Modelling Long-Range Context via Binary Partitioning

arXiv 1911.04070 ↗First posted 11 Nov 2019cs.CL cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

The Transformer model is widely successful on many natural language processing tasks. However, the quadratic complexity of self-attention limit its application on long text. In this paper, adopting a fine-to-coarse attention mechanism on multi-scale spans via binary partitioning (BP), we propose BP-Transformer (BPT for short). BPT yields O(k· nlog (n/k)) connections where k is a hyperparameter to control the density of attention. BPT has a good balance between computation complexity and model capacity. A series of experiments on text classification, machine translation and language modeling shows BPT has a superior performance for long text than previous self-attention models. Our code, hyperparameters and CUDA kernels for sparse attention are available in PyTorch.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 11 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 15; 2022: 6; 2023: 8; 2024: 5; 2025: 4.

38 papers.

Sort:

By area:

  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  2. 2021A Survey of Transformerscited by 401
  3. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  4. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 102
  5. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  6. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  7. 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
  8. 2021Memory-efficient Transformers via Top-k Attentioncited by 60
  9. 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 49
  10. 2021Long-Range Transformers for Dynamic Spatiotemporal Forecastingcited by 46
  11. 2025Log-Linear Attentioncited by 40
  12. 2024Foundation Models for Music: A Surveycited by 37
26 more
  1. 2021Hierarchical Learning for Generation with Long Source Sequencescited by 32
  2. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  3. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  4. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  5. 2022A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipeliningcited by 22
  6. 2024Extending LLMs' Context Window with 100 Samplescited by 21
  7. 2021On Learning the Transformer Kernelcited by 15
  8. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
  9. 2022Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputationcited by 12
  10. 2024Efficient Heterogeneous Large Language Model Decoding with Model-Attention Disaggregationcited by 12
  11. 2023Large Human Language Models: A Need and the Challengescited by 11
  12. 2024StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Lossescited by 9
  13. 2022Accelerating Attention through Gradient-Based Learned Runtime Pruningcited by 8
  14. 2021Value-aware Approximate Attentioncited by 5
  15. 2021What Helps Transformers Recognize Conversational Structure? Importance of Context, Punctuation, and Labels in Dialog Act Recognitioncited by 5
  16. 2023Infomaxformer: Maximum Entropy Transformer for Long Time-Series Forecasting Problemcited by 5
  17. 2021Sparse Fusion for Multimodal Transformerscited by 4
  18. 2023Efficient Beam Tree Recursioncited by 4
  19. 2021Adaptive Multi-Resolution Attention with Linear Complexitycited by 2
  20. 2022Efficient Representation Learning via Adaptive Context Poolingcited by 2
  21. 2023Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attentioncited by 2
  22. 2025Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problemscited by 1
  23. 2023A Quantitative Review on Language Model Efficiency Researchcited by 0
  24. 2023RCMHA: Relative Convolutional Multi-Head Attention for Natural Language Modellingcited by 0
  25. 2025ARLED: Leveraging LED-based ARMAN Model for Abstractive Summarization of Persian Long Documentscited by 0
  26. 2025Customizing the Inductive Biases of Softmax Attention using Structured Matricescited by 0