# Towards Efficient Multi-Scale Deformable Attention on NPU

arXiv: https://arxiv.org/abs/2505.14022 · HTML: https://mlsanity.com/paper/2505.14022/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 12 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 6 of 6)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [Visual In-Context Prompting](https://mlsanity.com/paper/2311.13601/) (2023)
- [DEFA: Efficient Deformable Attention Acceleration via Pruning-Assisted Grid-Sampling and Multi-Scale Parallel Processing](https://mlsanity.com/paper/2403.10913/) (2024)
- [MDHA: Multi-Scale Deformable Transformer with Hybrid Anchors for Multi-View 3D Object Detection](https://mlsanity.com/paper/2406.17654/) (2024)

## Built on by (most cited first, 1 of 1)

- [Vectorization Of Narrow Matrix Multiplication for Ascend AI Inference Acceleration](https://mlsanity.com/paper/2609.16009/) (2026)

## Abstract (arXiv)

Multi-scale deformable attention (MSDA) is a flexible and powerful feature extraction mechanism for visual tasks, but its random-access grid sampling strategy poses significant optimization challenges, especially on domain-specific accelerators such as NPUs. In this work, we present a co-design approach that systematically rethinks memory access and computation strategies for MSDA on the Ascend NPU architecture. With this co-design approach, our implementation supports both efficient forward and backward computation, is fully adapted for training workloads, and incorporates a suite of hardware-aware optimizations. Extensive experiments show that our solution achieves up to 5.9× (forward), 8.9× (backward), and 7.3× (end-to-end training) speedup over the grid sample-based baseline, and 1.9×, 2.4×, and 2.0× acceleration over the latest vendor library, respectively.