mlsanityData to 7 Oct 2026

Data Movement Is All You Need: A Case Study on Optimizing Transformers

arXiv 2007.00072 ↗First posted 30 Jun 2020cs.LG stat.MLcited by 108

Not on the map. Why?

Abstract (arXiv)

Transformers are one of the most important machine learning workloads today. Training one is a very compute-intensive task, often taking days or weeks, and significant attention has been given to optimizing transformers. Despite this, existing implementations do not efficiently utilize GPUs. We find that data movement is the key bottleneck when training. Due to Amdahl's Law and massive improvements in compute performance, training has now become memory-bound. Further, existing frameworks use suboptimal data layouts. Using these insights, we present a recipe for globally optimizing data movement in transformers. We reduce data movement by up to 22.91% and overall achieve a 1.30x performance improvement over state-of-the-art frameworks when training a BERT encoder layer and 1.19x for the entire BERT. Our approach is applicable more broadly to optimizing deep neural networks, and offers insight into how to tackle emerging performance bottlenecks.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 17; 2023: 22; 2024: 19; 2025: 17; 2026: 22.

108 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  3. 2023A Cookbook of Self-Supervised Learningcited by 248
  4. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  5. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  6. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  7. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  8. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  9. 2024SHViT: Single-Head Vision Transformer with Memory Efficient Macro Designcited by 55
  10. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  11. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 38
  12. 2023Reversible Vision Transformerscited by 33
38 more
  1. 2022SimA: Simple Softmax-free Attention for Vision Transformerscited by 32
  2. 2025Parallel Scaling Law for Language Modelscited by 29
  3. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelscited by 23
  4. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Modelscited by 19
  5. 2024SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Expertscited by 18
  6. 2021The CoRa Tensor Compiler: Compilation for Ragged Tensors with Minimal Paddingcited by 17
  7. 2023Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Uglycited by 17
  8. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 16
  9. 2024Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformerscited by 16
  10. 2021Clairvoyant Prefetching for Distributed Machine Learning I/Ocited by 14
  11. 2022Spatial Mixture-of-Expertscited by 14
  12. 2021FastSeq: Make Sequence Generation Fastercited by 13
  13. 2023VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Corescited by 13
  14. 2021Improving the Efficiency of Transformers for Resource-Constrained Devicescited by 12
  15. 2022A Comprehensive Study of Real-Time Object Detection Networks Across Multiple Domains: A Surveycited by 12
  16. 2024SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobilecited by 12
  17. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  18. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  19. 2024Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inferencecited by 9
  20. 2022A Comprehensive Study of Vision Transformers on Dense Prediction Taskscited by 8
  21. 2022HammingMesh: A Network Topology for Large-Scale Deep Learningcited by 8
  22. 2023Operator Fusion in XLA: Analysis and Evaluationcited by 8
  23. 2023Enabling Resource-efficient AIoT System with Cross-level Optimization: A surveycited by 8
  24. 2024Optimizing the Deployment of Tiny Transformers on Low-Power MCUscited by 8
  25. 2024FlashAttention on a Napkin: A Diagrammatic Approach to Deep Learning IO-Awarenesscited by 7
  26. 2022Survey: Exploiting Data Redundancy for Optimization of Deep Learningcited by 6
  27. 2023Dynamic Stashing Quantization for Efficient Transformer Trainingcited by 6
  28. 2025Tawa: Automatic Warp Specialization for Modern GPUs with Asynchronous Referencescited by 6
  29. 2026Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategiescited by 6
  30. 2023PolyThrottle: Energy-efficient Neural Network Inference on Edge Devicescited by 5
  31. 2024PIM-Opt: Demystifying Distributed Optimization Algorithms on a Real-World Processing-In-Memory Systemcited by 5
  32. 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activationcited by 5
  33. 2025Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEscited by 5
  34. 2026Retrieval-Aware Distillation for Transformer-SSM Hybridscited by 5
  35. 2026CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programscited by 5
  36. 2022Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reductioncited by 4
  37. 2021Demystifying BERT: Implications for Accelerator Designcited by 3
  38. 2021A Data-Centric Optimization Framework for Machine Learningcited by 3

50 most cited of 108 All 108 (JSON)