mlsanityData to 14 Oct 2026

DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training

arXiv 2310.03294 ↗First posted 5 Oct 2023cs.LG cs.AI cs.DCcited by 29

Not on the map. Why?

Abstract (arXiv)

FlashAttention (Dao, 2023) effectively reduces the quadratic peak memory usage to linear in training transformer-based large language models (LLMs) on a single GPU. In this paper, we introduce DISTFLASHATTN, a distributed memory-efficient attention mechanism optimized for long-context LLMs training. We propose three key techniques: token-level workload balancing, overlapping key-value communication, and a rematerialization-aware gradient checkpointing algorithm. We evaluate DISTFLASHATTN on Llama-7B and variants with sequence lengths from 32K to 512K. DISTFLASHATTN achieves 8x longer sequences, 4.45 - 5.64x speedup compared to Ring Self-Attention, 2 - 8x longer sequences, 1.24 - 2.01x speedup compared to Megatron-LM with FlashAttention. It achieves 1.67x and 1.26 - 1.88x speedup compared to recent Ring Attention and DeepSpeed-Ulysses. Code is available at https://github.com/RulinShao/LightSeq.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

19 of 25 references.

References · 19

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  11. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
7 more
  1. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  2. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  3. 2018Online normalizer calculation for softmaxcited by 162
  4. 2021Self-attention Does Not Need O(n2) Memorycited by 154
  5. 2022A Length-Extrapolatable Transformercited by 127
  6. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  7. 2021Multi-head or Single-head? An Empirical Comparison for Transformer Trainingcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 14; 2026: 9.

29 papers.

Sort:

By area:

  1. 2025Fast Video Generation with Sliding Tile Attentioncited by 128
  2. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  3. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 101
  4. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  5. 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
  6. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  7. 2025Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tilecited by 26
  8. 2024FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelismcited by 19
  9. 2023Efficient Parallelization Layouts for Large-Scale Distributed Model Trainingcited by 8
  10. 2025Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decodingcited by 7
  11. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contextscited by 4
  12. 2024Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignmentcited by 3
17 more
  1. 2025SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefillingcited by 3
  2. 2025Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequencescited by 3
  3. 2026LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budgetcited by 3
  4. 2025HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelismcited by 2
  5. 2025Pixel-Resolved Long-Context Learning for Turbulence at Exascale: Resolving Small-scale Eddies Toward the Viscous Limitcited by 1
  6. 2025Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networkscited by 1
  7. 2025A Scalable Distributed Framework for Multimodal GigaVoxel Image Registrationcited by 1
  8. 2026Efficient Scaling of LLM Training with Flexible Context Parallelismcited by 1
  9. 2025LV-XAttn: Distributed Cross-Attention for Long Visual Inputs in Multimodal Large Language Modelscited by 0
  10. 2025Scaling Generative Recommendations with Context Parallelism on Hierarchical Sequential Transducerscited by 0
  11. 2025TASP: Topology-aware Sequence Parallelismcited by 0
  12. 2026Data-Aware Random Feature Kernel for Transformerscited by 0
  13. 2026Stream-CQSA: Exact Out-of-Memory Recovery for Attentioncited by 0
  14. 2026ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformerscited by 0
  15. 2026The Life of a Token: from Words to Bits on the Wirecited by 0
  16. 2026FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attentioncited by 0
  17. 2026FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attentioncited by 0