# DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation

arXiv: https://arxiv.org/abs/1909.12471 · HTML: https://mlsanity.com/paper/1909.12471/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Sep 2019.

## Built on by (most cited first, 26 of 26)

- [A Survey on Deep Learning Technique for Video Segmentation](https://mlsanity.com/paper/2107.01153/) (2021)
- [Hierarchical Memory Matching Network for Video Object Segmentation](https://mlsanity.com/paper/2109.11404/) (2021)
- [SwiftNet: Real-time Video Object Segmentation](https://mlsanity.com/paper/2102.04604/) (2021)
- [Learning Position and Target Consistency for Memory-based Video Object Segmentation](https://mlsanity.com/paper/2104.04329/) (2021)
- [Breaking the "Object" in Video Object Segmentation](https://mlsanity.com/paper/2212.06200/) (2022)
- [Reliable Propagation-Correction Modulation for Video Object Segmentation](https://mlsanity.com/paper/2112.02853/) (2021)
- [Differentiable Multi-Granularity Human Representation Learning for Instance-Aware Human Semantic Parsing](https://mlsanity.com/paper/2103.04570/) (2021)
- [TrackMPNN: A Message Passing Graph Neural Architecture for Multi-Object Tracking](https://mlsanity.com/paper/2101.04206/) (2021)
- [HODOR: High-level Object Descriptors for Object Re-segmentation in Video Learned from Static Images](https://mlsanity.com/paper/2112.09131/) (2021)
- [D^2Conv3D: Dynamic Dilated Convolutions for Object Segmentation in Videos](https://mlsanity.com/paper/2111.07774/) (2021)
- [Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild](https://mlsanity.com/paper/2103.10391/) (2021)
- [Tackling Background Distraction in Video Object Segmentation](https://mlsanity.com/paper/2207.06953/) (2022)
- [Towards Robust Video Object Segmentation with Adaptive Object Calibration](https://mlsanity.com/paper/2207.00887/) (2022)
- [Combinatorial Optimization for Panoptic Segmentation: A Fully Differentiable Approach](https://mlsanity.com/paper/2106.03188/) (2021)
- [Co-attention Propagation Network for Zero-Shot Video Object Segmentation](https://mlsanity.com/paper/2304.03910/) (2023)
- [VideoClick: Video Object Segmentation with a Single Click](https://mlsanity.com/paper/2101.06545/) (2021)
- [Semi-Supervised Video Inpainting with Cycle Consistency Constraints](https://mlsanity.com/paper/2208.06807/) (2022)
- [MUNet: Motion Uncertainty-aware Semi-supervised Video Object Segmentation](https://mlsanity.com/paper/2111.14646/) (2021)
- [GLAN: A Graph-based Linear Assignment Network](https://mlsanity.com/paper/2201.02057/) (2022)
- [Fast Pixel-Matching for Video Object Segmentation](https://mlsanity.com/paper/2107.04279/) (2021)
- [Fast and Resource-Efficient Object Tracking on Edge Devices: A Measurement Study](https://mlsanity.com/paper/2309.02666/) (2023)
- [Exploring the Semi-supervised Video Object Segmentation Problem from a Cyclic Perspective](https://mlsanity.com/paper/2111.01323/) (2021)
- [FlowVOS: Weakly-Supervised Visual Warping for Detail-Preserving and Temporally Consistent Single-Shot Video Object Segmentation](https://mlsanity.com/paper/2111.10621/) (2021)
- [Pixel-Level Equalized Matching for Video Object Segmentation](https://mlsanity.com/paper/2209.03139/) (2022)
- [How to Backpropagate through Hungarian in Your DETR?](https://mlsanity.com/paper/2211.14448/) (2022)
- [Reliability-Hierarchical Memory Network for Scribble-Supervised Video Object Segmentation](https://mlsanity.com/paper/2303.14384/) (2023)

## Abstract (arXiv)

In this paper, we propose the differentiable mask-matching network (DMM-Net) for solving the video object segmentation problem where the initial object masks are provided. Relying on the Mask R-CNN backbone, we extract mask proposals per frame and formulate the matching between object templates and proposals at one time step as a linear assignment problem where the cost matrix is predicted by a CNN. We propose a differentiable matching layer by unrolling a projected gradient descent algorithm in which the projection exploits the Dykstra's algorithm. We prove that under mild conditions, the matching is guaranteed to converge to the optimum. In practice, it performs similarly to the Hungarian algorithm during inference. Meanwhile, we can back-propagate through it to learn the cost matrix. After matching, a refinement head is leveraged to improve the quality of the matched mask. Our DMM-Net achieves competitive results on the largest video object segmentation dataset YouTube-VOS. On DAVIS 2017, DMM-Net achieves the best performance without online learning on the first frames. Without any fine-tuning, DMM-Net performs comparably to state-of-the-art methods on SegTrack v2 dataset. At last, our matching layer is very simple to implement; we attach the PyTorch code (<50 lines) in the supplementary material. Our code is released at https://github.com/ZENGXH/DMMNet.