# D^2Conv3D: Dynamic Dilated Convolutions for Object Segmentation in Videos

arXiv: https://arxiv.org/abs/2111.07774 · HTML: https://mlsanity.com/paper/2111.07774/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 54 references that are in the map.
- First posted on arXiv 15 Nov 2021.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Deformable ConvNets v2: More Deformable, Better Results](https://mlsanity.com/paper/1811.11168/) (2018)
- [HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking](https://mlsanity.com/paper/2009.07736/) (2020)
- [End-to-End Video Instance Segmentation with Transformers](https://mlsanity.com/paper/2011.14503/) (2020)
- [Video Instance Segmentation](https://mlsanity.com/paper/1905.04804/) (2019)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [MOTS: Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/1902.03604/) (2019)
- [FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation](https://mlsanity.com/paper/1902.09513/) (2019)
- [Learning Depth-Guided Convolutions for Monocular 3D Object Detection](https://mlsanity.com/paper/1912.04799/) (2019)
- [See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networks](https://mlsanity.com/paper/2001.06810/) (2020)
- [Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagation](https://mlsanity.com/paper/1912.04573/) (2019)
- [STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos](https://mlsanity.com/paper/2003.08429/) (2020)
- [SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentation](https://mlsanity.com/paper/2007.14772/) (2020)
- [Efficient Video Object Segmentation via Network Modulation](https://mlsanity.com/paper/1802.01218/) (2018)
- [The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation](https://mlsanity.com/paper/1905.00737/) (2019)
- [RVOS: End-to-End Recurrent Network for Video Object Segmentation](https://mlsanity.com/paper/1903.05612/) (2019)

## Built on by (most cited first, 20 of 20)

- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Treating Motion as Option to Reduce Motion Dependency in Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2209.03138/) (2022)
- [Unsupervised Video Object Segmentation via Prototype Memory Network](https://mlsanity.com/paper/2209.03712/) (2022)
- [Bootstrapping Objectness from Videos by Relaxed Common Fate and Visual Grouping](https://mlsanity.com/paper/2304.08025/) (2023)
- [Motion-inductive Self-supervised Object Discovery in Videos](https://mlsanity.com/paper/2210.00221/) (2022)
- [Dual Prototype Attention for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2211.12036/) (2022)
- [Co-attention Propagation Network for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2304.03910/) (2023)
- [SMITE: Segment Me In TimE](https://mlsanity.com/paper/2410.18538/) (2024)
- [FE-Adapter: Adapting Image-based Emotion Classifiers to Videos](https://mlsanity.com/paper/2408.02421/) (2024)
- [Tsanet: Temporal and Scale Alignment for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2303.04376/) (2023)
- [Hierarchical Graph Pattern Understanding for Zero-Shot VOS](https://mlsanity.com/paper/2312.09525/) (2023)
- [Improving Unsupervised Video Object Segmentation via Fake Flow Generation](https://mlsanity.com/paper/2407.11714/) (2024)
- [Improving Unsupervised Video Object Segmentation with Motion-Appearance Synergy](https://mlsanity.com/paper/2212.08816/) (2022)
- [Treating Motion as Option with Output Selection for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2309.14786/) (2023)
- [Attentive Dilated Convolution for Automatic Sleep Staging using Force-directed Layout](https://mlsanity.com/paper/2409.01962/) (2024)
- [MovSAM: A Single-image Moving Object Segmentation Framework Based on Deep Thinking](https://mlsanity.com/paper/2504.06863/) (2025)
- [DepthFlow: Exploiting Depth-Flow Structural Correlations for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2507.19790/) (2025)
- [Spatio-Temporal Attention for Consistent Video Semantic Segmentation in Automated Driving](https://mlsanity.com/paper/2602.10052/) (2026)
- [CMTM: Cross-Modal Token Modulation for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2604.14630/) (2026)
- [Adaptive 3D Convolution for Remote Sensing Image Fusion](https://mlsanity.com/paper/2605.09455/) (2026)

## Abstract (arXiv)

Despite receiving significant attention from the research community, the task of segmenting and tracking objects in monocular videos still has much room for improvement. Existing works have simultaneously justified the efficacy of dilated and deformable convolutions for various image-level segmentation tasks. This gives reason to believe that 3D extensions of such convolutions should also yield performance improvements for video-level segmentation tasks. However, this aspect has not yet been explored thoroughly in existing literature. In this paper, we propose Dynamic Dilated Convolutions (D^2Conv3D): a novel type of convolution which draws inspiration from dilated and deformable convolutions and extends them to the 3D (spatio-temporal) domain. We experimentally show that D^2Conv3D can be used to improve the performance of multiple 3D CNN architectures across multiple video segmentation related benchmarks by simply employing D^2Conv3D as a drop-in replacement for standard convolutions. We further show that D^2Conv3D out-performs trivial extensions of existing dilated and deformable convolutions to 3D. Lastly, we set a new state-of-the-art on the DAVIS 2016 Unsupervised Video Object Segmentation benchmark. Code is made publicly available at https://github.com/Schmiddo/d2conv3d .