# Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications

arXiv: https://arxiv.org/abs/2401.06197 · HTML: https://mlsanity.com/paper/2401.06197/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 48 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)

## Built on by (most cited first, 30 of 50)

- [Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures](https://mlsanity.com/paper/2403.02308/) (2024)
- [Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera](https://mlsanity.com/paper/2501.02464/) (2025)
- [ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictions](https://mlsanity.com/paper/2411.07725/) (2024)
- [Frequency Dynamic Convolution for Dense Image Prediction](https://mlsanity.com/paper/2503.18783/) (2025)
- [DefMamba: Deformable Visual State Space Model](https://mlsanity.com/paper/2504.05794/) (2025)
- [Open Panoramic Segmentation](https://mlsanity.com/paper/2407.02685/) (2024)
- [FlowDCN: Exploring DCN-like Architectures for Fast Image Generation with Arbitrary Resolution](https://mlsanity.com/paper/2410.22655/) (2024)
- [TimePro: Efficient Multivariate Long-term Time Series Forecasting with Variable- and Time-Aware Hyper-state](https://mlsanity.com/paper/2505.20774/) (2025)
- [DehazeDCT: Towards Effective Non-Homogeneous Dehazing via Deformable Convolutional Transformer](https://mlsanity.com/paper/2407.05169/) (2024)
- [Diff2I2P: Differentiable Image-to-Point Cloud Registration with Diffusion Prior](https://mlsanity.com/paper/2507.06651/) (2025)
- [UniDAC: Universal Metric Depth Estimation for Any Camera](https://mlsanity.com/paper/2603.27105/) (2026)
- [Deformable Mamba for Wide Field of View Segmentation](https://mlsanity.com/paper/2411.16481/) (2024)
- [Learning Modality-Aware Representations: Adaptive Group-wise Interaction Network for Multimodal MRI Synthesis](https://mlsanity.com/paper/2411.14684/) (2024)
- [3rd Workshop on Maritime Computer Vision (MaCVi) 2025: Challenge Results](https://mlsanity.com/paper/2501.10343/) (2025)
- [DynRefer: Delving into Region-level Multimodal Tasks via Dynamic Resolution](https://mlsanity.com/paper/2405.16071/) (2024)
- [XS-VID: A Large-Scale Benchmark for Small Object Detection and Tracking in Videos](https://mlsanity.com/paper/2407.18137/) (2024)
- [UV-Mamba: A DCN-Enhanced State Space Model for Urban Village Boundary Identification in High-Resolution Remote Sensing Images](https://mlsanity.com/paper/2409.03431/) (2024)
- [UniConvNet: Expanding Effective Receptive Field while Maintaining Asymptotically Gaussian Distribution for ConvNets of Any Scale](https://mlsanity.com/paper/2508.09000/) (2025)
- [Looking Locally: Object-Centric Vision Transformers as Foundation Models for Efficient Segmentation](https://mlsanity.com/paper/2502.02763/) (2025)
- [VideoSAM: A Large Vision Foundation Model for High-Speed Video Segmentation](https://mlsanity.com/paper/2410.21304/) (2024)
- [Technical Report for ICRA 2025 GOOSE 2D Semantic Segmentation Challenge: Boosting Off-Road Segmentation via Photometric Distortion and Exponential Moving Average](https://mlsanity.com/paper/2505.11769/) (2025)
- [Uni-LVC: A Unified Method for Intra- and Inter-Mode Learned Video Compression](https://mlsanity.com/paper/2603.05756/) (2026)
- [DepthAgent: Towards Better Universal Depth Estimation via Sample-wise Expert Selection](https://mlsanity.com/paper/2605.23281/) (2026)
- [Bend the Basics: Degradation-Aware Deformable Tokenization for All-in-One Image Restoration](https://mlsanity.com/paper/2608.06832/) (2026)
- [Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution](https://mlsanity.com/paper/2409.17597/) (2024)
- [Efficient High-Resolution Visual Representation Learning with State Space Model for Human Pose Estimation](https://mlsanity.com/paper/2410.03174/) (2024)
- [CoMiX: Cross-Modal Fusion with Deformable Convolutions for HSI-X Semantic Segmentation](https://mlsanity.com/paper/2411.09023/) (2024)
- [Conformable Convolution for Topologically Aware Learning of Complex Anatomical Structures](https://mlsanity.com/paper/2412.20608/) (2024)
- [KunPeng: A Global Ocean Environmental Model](https://mlsanity.com/paper/2504.04766/) (2025)
- [Deformable Convolution Module with Globally Learned Relative Offsets for Fundus Vessel Segmentation](https://mlsanity.com/paper/2507.18354/) (2025)

## Abstract (arXiv)

We introduce Deformable Convolution v4 (DCNv4), a highly efficient and effective operator designed for a broad spectrum of vision applications. DCNv4 addresses the limitations of its predecessor, DCNv3, with two key enhancements: 1. removing softmax normalization in spatial aggregation to enhance its dynamic property and expressive power and 2. optimizing memory access to minimize redundant operations for speedup. These improvements result in a significantly faster convergence compared to DCNv3 and a substantial increase in processing speed, with DCNv4 achieving more than three times the forward speed. DCNv4 demonstrates exceptional performance across various tasks, including image classification, instance and semantic segmentation, and notably, image generation. When integrated into generative models like U-Net in the latent diffusion model, DCNv4 outperforms its baseline, underscoring its possibility to enhance generative models. In practical applications, replacing DCNv3 with DCNv4 in the InternImage model to create FlashInternImage results in up to 80% speed increase and further performance improvement without further modifications. The advancements in speed and efficiency of DCNv4, combined with its robust performance across diverse vision tasks, show its potential as a foundational building block for future vision models.