# Making a Case for 3D Convolutions for Object Segmentation in Videos

arXiv: https://arxiv.org/abs/2008.11516 · HTML: https://mlsanity.com/paper/2008.11516/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2020.

## Built on by (most cited first, 30 of 56)

- [Audio-Visual Segmentation](https://mlsanity.com/paper/2207.05042/) (2022)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [Full-Duplex Strategy for Video Object Segmentation](https://mlsanity.com/paper/2108.03151/) (2021)
- [Audio-Visual Segmentation with Semantics](https://mlsanity.com/paper/2301.13190/) (2023)
- [AVSegFormer: Audio-Visual Segmentation with Transformer](https://mlsanity.com/paper/2307.01146/) (2023)
- [BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video](https://mlsanity.com/paper/2209.12118/) (2022)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation](https://mlsanity.com/paper/2305.16318/) (2023)
- [CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation](https://mlsanity.com/paper/2309.09709/) (2023)
- [A Unified Transformer Framework for Group-based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection](https://mlsanity.com/paper/2203.04708/) (2022)
- [Annotation-free Audio-Visual Segmentation](https://mlsanity.com/paper/2305.11019/) (2023)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2207.08485/) (2022)
- [Improving Audio-Visual Segmentation with Bidirectional Generation](https://mlsanity.com/paper/2308.08288/) (2023)
- [Treating Motion as Option to Reduce Motion Dependency in Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2209.03138/) (2022)
- [Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation](https://mlsanity.com/paper/2307.13236/) (2023)
- [Contrastive Conditional Latent Diffusion for Audio-visual Segmentation](https://mlsanity.com/paper/2307.16579/) (2023)
- [Unsupervised Video Object Segmentation via Prototype Memory Network](https://mlsanity.com/paper/2209.03712/) (2022)
- [Weakly Supervised Video Salient Object Detection](https://mlsanity.com/paper/2104.02391/) (2021)
- [Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation](https://mlsanity.com/paper/2312.06462/) (2023)
- [D^2Conv3D: Dynamic Dilated Convolutions for Object Segmentation in Videos](https://mlsanity.com/paper/2111.07774/) (2021)
- [FUN-SIS: a Fully UNsupervised approach for Surgical Instrument Segmentation](https://mlsanity.com/paper/2202.08141/) (2022)
- [QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition](https://mlsanity.com/paper/2310.00132/) (2023)
- [Visual Semantic Segmentation Based on Few/Zero-Shot Learning: An Overview](https://mlsanity.com/paper/2211.08352/) (2022)
- [A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2209.09341/) (2022)
- [Bootstrapping Objectness from Videos by Relaxed Common Fate and Visual Grouping](https://mlsanity.com/paper/2304.08025/) (2023)
- [Dual Prototype Attention for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2211.12036/) (2022)
- [SimulFlow: Simultaneously Extracting Feature and Identifying Target for Unsupervised Video Object Segmentation](https://mlsanity.com/paper/2311.18286/) (2023)
- [Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation](https://mlsanity.com/paper/2308.06693/) (2023)
- [Panoramic Video Salient Object Detection with Ambisonic Audio Guidance](https://mlsanity.com/paper/2211.14419/) (2022)

## Abstract (arXiv)

The task of object segmentation in videos is usually accomplished by processing appearance and motion information separately using standard 2D convolutional networks, followed by a learned fusion of the two sources of information. On the other hand, 3D convolutional networks have been successfully applied for video classification tasks, but have not been leveraged as effectively to problems involving dense per-pixel interpretation of videos compared to their 2D convolutional counterparts and lag behind the aforementioned networks in terms of performance. In this work, we show that 3D CNNs can be effectively applied to dense video prediction tasks such as salient object segmentation. We propose a simple yet effective encoder-decoder network architecture consisting entirely of 3D convolutions that can be trained end-to-end using a standard cross-entropy loss. To this end, we leverage an efficient 3D encoder, and propose a 3D decoder architecture, that comprises novel 3D Global Convolution layers and 3D Refinement modules. Our approach outperforms existing state-of-the-arts by a large margin on the DAVIS'16 Unsupervised, FBMS and ViSal dataset benchmarks in addition to being faster, thus showing that our architecture can efficiently learn expressive spatio-temporal features and produce high quality video segmentation masks. We have made our code and trained models publicly available at https://github.com/sabarim/3DC-Seg.