# DEFA: Efficient Deformable Attention Acceleration via Pruning-Assisted Grid-Sampling and Multi-Scale Parallel Processing

arXiv: https://arxiv.org/abs/2403.10913 · HTML: https://mlsanity.com/paper/2403.10913/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2024.

## Built on by (most cited first, 3 of 3)

- [Towards Efficient Multi-Scale Deformable Attention on NPU](https://mlsanity.com/paper/2505.14022/) (2025)
- [A Memory-Efficient Framework for Deformable Transformer with Neural Architecture Search](https://mlsanity.com/paper/2507.11549/) (2025)
- [QUILL: An Algorithm-Architecture Co-Design for Cache-Local Deformable Attention](https://mlsanity.com/paper/2511.13679/) (2025)

## Abstract (arXiv)

Multi-scale deformable attention (MSDeformAttn) has emerged as a key mechanism in various vision tasks, demonstrating explicit superiority attributed to multi-scale grid-sampling. However, this newly introduced operator incurs irregular data access and enormous memory requirement, leading to severe PE underutilization. Meanwhile, existing approaches for attention acceleration cannot be directly applied to MSDeformAttn due to lack of support for this distinct procedure. Therefore, we propose a dedicated algorithm-architecture co-design dubbed DEFA, the first-of-its-kind method for MSDeformAttn acceleration. At the algorithm level, DEFA adopts frequency-weighted pruning and probability-aware pruning for feature maps and sampling points respectively, alleviating the memory footprint by over 80%. At the architecture level, it explores the multi-scale parallelism to boost the throughput significantly and further reduces the memory access via fine-grained layer fusion and feature map reusing. Extensively evaluated on representative benchmarks, DEFA achieves 10.1-31.9x speedup and 20.3-37.7x energy efficiency boost compared to powerful GPUs. It also rivals the related accelerators by 2.2-3.7x energy efficiency improvement while providing pioneering support for MSDeformAttn.