mlsanityData to 14 Oct 2026

FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks

arXiv 2107.06419 ↗First posted 13 Jul 2021cs.LG cs.ARcited by 39

Not on the map. Why?

Abstract (arXiv)

Attention mechanisms, primarily designed to capture pairwise correlations between words, have become the backbone of machine learning, expanding beyond natural language processing into other domains. This growth in adaptation comes at the cost of prohibitively large memory requirements and computational complexity, especially at higher number of input elements. This limitation is due to inherently limited data reuse opportunities and quadratic growth in memory footprints, leading to severe memory-boundedness and limited scalability of input elements. This work addresses these challenges by devising a tailored dataflow optimization, called FLAT, for attention mechanisms without altering their functionality. This dataflow processes costly attention operations through a unique fusion mechanism, transforming the memory footprint quadratic growth to merely a linear one. To realize the full potential of this bespoke mechanism, we propose a tiling approach to enhance the data reuse across attention operations. Our method both mitigates the off-chip bandwidth bottleneck as well as reduces the on-chip memory requirement. FLAT delivers 1.94x (1.76x) speedup and 49% and (42%) of energy savings compared to the state-of-the-art Edge (Cloud) accelerators with no customized dataflow optimization. When on-chip resources are scarce (20 KB-200 KB), FLAT yields, on average, 1.5x end-to-end latency reduction across a diverse range of conventional attention-based models with input sequence lengths ranging from 512-token to 64K-token. Our evaluations demonstrate that state-of-the-art DNN dataflow applied to attention operations reach the efficiency limit for inputs above 512 elements. In contrast, FLAT unblocks transformer models for inputs with up to 64K elements

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 59 of its 111 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

59 of 111 references.

References · 59

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2020Rethinking Attention with Performerscited by 1,601
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
38 more
  1. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  5. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  6. 2018Image Transformercited by 772
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  8. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  9. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  11. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  12. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  13. 2020TransTrack: Multiple Object Tracking with Transformercited by 371
  14. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  15. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  16. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  17. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  18. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  19. 2018Music Transformercited by 211
  20. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  21. 2020Synthesizer: Rethinking Self-Attention in Transformer Modelscited by 205
  22. 2019Structured Pruning of Large Language Modelscited by 203
  23. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  24. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  25. 2019Adaptively Sparse Transformerscited by 155
  26. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  27. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  28. 2018Learning to Optimize Tensor Programscited by 140
  29. 2021Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphscited by 110
  30. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  31. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  32. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 101
  33. 2020HyGCN: A GCN Accelerator with Hybrid Architecturecited by 74
  34. 2018Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Codecited by 53
  35. 2020Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformerscited by 49
  36. 2021An Evaluation of Edge TPU Accelerators for Convolutional Neural Networkscited by 39
  37. 2020GRIP: A Graph Neural Network Accelerator Architecturecited by 37
  38. 2018Interstellar: Using Halide's Scheduling Language to Analyze DNN Acceleratorscited by 34

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 5; 2024: 13; 2025: 13; 2026: 7.

39 papers.

Sort:

By area:

  1. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  3. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  4. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 64
  5. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
  6. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batchingcited by 34
  7. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  8. 2023Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Productioncited by 21
  9. 2025PAPI: Exploiting Dynamic Parallelism in Large Language Model Decoding with a Processing-In-Memory-Enabled Computing Systemcited by 20
  10. 2024ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameterscited by 14
  11. 2022Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputationcited by 12
  12. 2023Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Trainingcited by 11
27 more
  1. 2024When Linear Attention Meets Autoregressive Decoding: Towards More Effective and Efficient Linearized Large Language Modelscited by 9
  2. 2025QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edgecited by 8
  3. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  4. 2024Understanding the Performance Horizon of the Latest ML Workloads with NonGEMM Workloadscited by 4
  5. 2024Characterizing the Accuracy -- Efficiency Trade-off of Low-rank Decomposition in Language Modelscited by 4
  6. 2025UbiMoE: A Ubiquitous Mixture-of-Experts Vision Transformer Accelerator With Hybrid Computation Pattern on FPGAcited by 4
  7. 2023AttMEMO : Accelerating Transformers with Memoization on Big Memory Systemscited by 3
  8. 2025SystolicAttention: Fusing FlashAttention within a Single Systolic Arraycited by 3
  9. 2023A Dataflow Compiler for Efficient LLM Inference using Custom Microscaling Formatscited by 2
  10. 2025Architecting Long-Context LLM Acceleration with Packing-Prefetch Scheduler and Ultra-Large Capacity On-Chip Memoriescited by 2
  11. 2023Subgraph Stationary Hardware-Software Inference Co-Designcited by 1
  12. 2025Beyond Moore's Law: Harnessing the Redshift of Generative AI with Effective Hardware-Software Co-Designcited by 1
  13. 2025AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Designcited by 1
  14. 2025A Study on Inference Latency for Vision Transformers on Mobile Devicescited by 1
  15. 2026VersaQ-3D: Architecture Support for Visual Geometry Grounded Transformers via Versatile Quantizationcited by 1
  16. 2024Comprehensive Performance Modeling and System Design Insights for Foundation Modelscited by 0
  17. 2024FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Accelerationcited by 0
  18. 2024MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devicescited by 0
  19. 2025Accelerating Sparse Transformer Inference on GPUcited by 0
  20. 2025DCO: Dynamic Cache Orchestration for LLM Accelerators through Predictive Managementcited by 0
  21. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0
  22. 2026From Buffers to Registers: Unlocking Fine-Grained FlashAttention with Hybrid-Bonded 3D NPU Co-Designcited by 0
  23. 2026RedFuser: An Automatic Operator Fusion Framework for Cascaded Reductions on AI Acceleratorscited by 0
  24. 2026LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Modelscited by 0
  25. 2026FastTPS: An Optimized Method for LLM Token Phase for AI acceleratorscited by 0
  26. 2026Transformer Accelerator (TFA): A Macro-Op INT8 Hardware Chip for Transformer Inference and Machine Translationcited by 0
  27. 2026PATTON: Enabling Commodity PIM for Production LLM Servingcited by 0