Towards Efficient Multi-Scale Deformable Attention on NPU
Not on the map. Why?
Abstract (arXiv)
Multi-scale deformable attention (MSDA) is a flexible and powerful feature extraction mechanism for visual tasks, but its random-access grid sampling strategy poses significant optimization challenges, especially on domain-specific accelerators such as NPUs. In this work, we present a co-design approach that systematically rethinks memory access and computation strategies for MSDA on the Ascend NPU architecture. With this co-design approach, our implementation supports both efficient forward and backward computation, is fully adapted for training workloads, and incorporates a suite of hardware-aware optimizations. Extensive experiments show that our solution achieves up to 5.9× (forward), 8.9× (backward), and 7.3× (end-to-end training) speedup over the grid sample-based baseline, and 1.9×, 2.4×, and 2.0× acceleration over the latest vendor library, respectively.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 6 of its 12 references that are in the map. First posted on arXiv 20 May 2025.
Builds on
6 of 12 references.
References · 6
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
- 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
- 2023Visual In-Context Promptingcited by 52
- 2024DEFA: Efficient Deformable Attention Acceleration via Pruning-Assisted Grid-Sampling and Multi-Scale Parallel Processingcited by 3
- 2024MDHA: Multi-Scale Deformable Transformer with Hybrid Anchors for Multi-View 3D Object Detectioncited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.