mlsanityData to 7 Oct 2026

Blockwise Parallel Transformer for Large Context Models

arXiv 2305.19370 ↗First posted 30 May 2023cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Transformers have emerged as the cornerstone of state-of-the-art natural language processing models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands posed by the self-attention mechanism and the large feedforward network in Transformers limit their ability to handle long sequences, thereby creating challenges for tasks involving multiple long sequences or long-term dependencies. We present a distinct approach, Blockwise Parallel Transformer (BPT), that leverages blockwise computation of self-attention and feedforward network fusion to minimize memory costs. By processing longer input sequences while maintaining memory efficiency, BPT enables training sequences 32 times longer than vanilla Transformers and up to 4 times longer than previous memory-efficient methods. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of BPT in reducing memory requirements and improving performance.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 51 of its 61 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

51 of 61 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  9. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  11. 2023Teaching Large Language Models to Self-Debugcited by 963
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  13. 2020Efficient Transformers: A Surveycited by 784
  14. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  15. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  16. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  17. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  18. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  19. 2019Axial Attention in Multidimensional Transformerscited by 350
  20. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  21. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  22. 2022Transformer Quality in Linear Timecited by 189
  23. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  24. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  25. 2018Online normalizer calculation for softmaxcited by 154
  26. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  27. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  28. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  29. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  30. 2021An Attention Free Transformercited by 111
  31. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 109
  32. 2021LambdaNetworks: Modeling Long-Range Interactions Without Attentioncited by 107
  33. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  34. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  35. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  36. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  37. 2022Pretraining Without Attentioncited by 51
  38. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 6; 2026: 1.

12 papers.

  1. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  2. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  3. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  4. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  5. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  6. 2024RetinaGS: Scalable Training for Dense Scene Rendering with Billion-Scale 3D Gaussianscited by 16
  7. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 9
  8. 2024CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novelscited by 2
  9. 2025Hierarchical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLMcited by 2
  10. 2025Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibrationcited by 1
  11. 2026Kaczmarz Linear Attentioncited by 1
  12. 2025Hierarchical Sparse Plus Low Rank Compression of LLMcited by 0