mlsanityData to 14 Oct 2026

SALO: An Efficient Spatial Accelerator Enabling Hybrid Sparse Attention Mechanisms for Long Sequences

arXiv 2206.14550 ↗First posted 29 Jun 2022cs.AR cs.AI cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

The attention mechanisms of transformers effectively extract pertinent information from the input sequence. However, the quadratic complexity of self-attention w.r.t the sequence length incurs heavy computational and memory burdens, especially for tasks with long sequences. Existing accelerators face performance degradation in these tasks. To this end, we propose SALO to enable hybrid sparse attention mechanisms for long sequences. SALO contains a data scheduler to map hybrid sparse attention patterns onto hardware and a spatial accelerator to perform the efficient attention computation. We show that SALO achieves 17.66x and 89.33x speedup on average compared to GPU and CPU implementations, respectively, on typical workloads, i.e., Longformer and ViL.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 14 of its 17 references that are in the map. First posted on arXiv 29 Jun 2022.

Builds on

14 of 17 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  8. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  9. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  10. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 101
  11. 2019Star-Transformercited by 94
  12. 2020FTRANS: Energy-Efficient Acceleration of Transformers using FPGAcited by 54
2 more
  1. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 54
  2. 2019QPyTorch: A Low-Precision Arithmetic Simulation Frameworkcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 4; 2025: 1; 2026: 1.

10 papers.

  1. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 80
  2. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  3. 2022ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attentioncited by 29
  4. 2024SWAT: Scalable and Efficient Window Attention-based Transformers Acceleration on FPGAscited by 6
  5. 2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitivenesscited by 4
  6. 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Servingcited by 3
  7. 2024Hardware Acceleration of LLMs: A comprehensive survey and comparisoncited by 3
  8. 2023YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUscited by 1
  9. 2024FAMOUS: Flexible Accelerator for the Attention Mechanism of Transformer on UltraScale+ FPGAscited by 1
  10. 2026ELiTeFormer: An Efficient Transformer for FPGAscited by 0