mlsanityData to 14 Oct 2026

MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devices

arXiv 2411.17720 ↗First posted 20 Nov 2024cs.DC cs.AI cs.PFcited by 0

Not on the map. Why?

Abstract (arXiv)

The advent of foundation models have revolutionized various fields, enabling unprecedented task accuracy and flexibility in computational linguistics, computer vision and other domains. Attention mechanism has become an essential component of foundation models, due to their superb capability of capturing correlations in a sequence. However, attention results in quadratic complexity in memory and compute as the context length grows. Although many fusion-based exact attention acceleration algorithms have been developed for datacenter-grade GPUs and accelerators leveraging multi-core parallelism and data locality, yet it remains a significant challenge to accelerate attention on resource-constrained edge neural accelerators with limited compute units and stringent on-chip caches. In this paper, we propose a scheme for exact attention inference acceleration on memory-constrained edge accelerators, by parallelizing the utilization of heterogeneous compute units, i.e., vector processing units and matrix processing units. Our method involves scheduling workloads onto these different compute units in a multi-tiered tiling scheme to process tiled vector workloads and matrix workloads in attention as two streams, respecting the workload dependencies. We search for tiling factors to maximize the parallelization of both compute units while considering I/O overhead, and propose a proactive cache overwrite strategy to avoid undesirable cache spills in reality. Extensive results based on open-sourced simulation frameworks show up to 2.75x speedup and 54% reduction in energy consumption as compared to the state-of-the-art attention fusion method (FLAT) in the edge computing scenario. Further experiments on a real-world edge neural processing unit demonstrate speedup of up to 1.76x for attention as compared to FLAT, without affecting model output accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 51 of its 79 references that are in the map. First posted on arXiv 20 Nov 2024.

Builds on

51 of 79 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022Scalable Diffusion Models with Transformerscited by 6,054
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2020Reformer: The Efficient Transformercited by 1,668
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  10. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  11. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  13. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  14. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  15. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
  16. 2021Post-Training Quantization for Vision Transformercited by 278
  17. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  18. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  19. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  20. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  21. 2018Learning to Optimize Tensor Programscited by 140
  22. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  23. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 108
  24. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 96
  25. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 95
  26. 2022Unified Visual Transformer Compressioncited by 79
  27. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 71
  28. 2021DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusioncited by 57
  29. 2018Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Codecited by 53
  30. 2023Knowledge Diffusion for Distillationcited by 53
  31. 2024Progressive Knowledge Distillation Of Stable Diffusion XL Using Layer Level Losscited by 52
  32. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 39
  33. 2021DKM: Differentiable K-Means Clustering Layer for Neural Network Compressioncited by 31
  34. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  35. 2021Improving the Efficiency of Transformers for Resource-Constrained Devicescited by 12
  36. 2022DeFiNES: Enabling Fast Exploration of the Depth-first Scheduling Space for DNN Accelerators through Analytical Modelingcited by 12
  37. 2023oneDNN Graph Compiler: A Hybrid Approach for High-Performance Deep Learning Compilationcited by 7
  38. 2024KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generationcited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026FlatAttention: Dataflow and Fabric Collectives Co-Optimization for Large Attention-Based Model Inference on Tile-Based Accelerators
  2. 2026SwiftKV: An Edge-Oriented Attention Algorithm and Multi-Head Accelerator for Fast, Efficient LLM Decoding
  3. 2025IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference
  4. 2026AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
  5. 2026STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU
  6. 2025Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention
  7. 2025SLIM: A Heterogeneous Accelerator for Edge Inference of Sparse Large Language Model via Adaptive Thresholding
  8. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference