mlsanityData to 7 Oct 2026

A Survey on Deep Learning Technique for Video Segmentation

arXiv 2107.01153 ↗First posted 2 Jul 2021cs.CVcited by 132

Not on the map. Why?

Abstract (arXiv)

Video segmentation -- partitioning video frames into multiple segments or objects -- plays a critical role in a broad range of practical applications, from enhancing visual effects in movie, to understanding scenes in autonomous driving, to creating virtual background in video conferencing. Recently, with the renaissance of connectionism in computer vision, there has been an influx of deep learning based approaches for video segmentation that have delivered compelling performance. In this survey, we comprehensively review two basic lines of research -- generic object segmentation (of unknown categories) in videos, and video semantic segmentation -- by introducing their respective task settings, background concepts, perceived need, development history, and main challenges. We also offer a detailed overview of representative literature on both methods and datasets. We further benchmark the reviewed methods on several well-known datasets. Finally, we point out open issues in this field, and suggest opportunities for further research. We also provide a public website to continuously track developments in this fast advancing field: https://github.com/tfzhou/VS-Survey.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Builds on 144 of its 274 references that are in the map. First posted on arXiv 2 Jul 2021.

Builds on

144 of 274 references.

References · 144

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2017Mask R-CNNcited by 7,259
  5. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,282
  9. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  10. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  11. 2017Localizing Moments in Video with Natural Languagecited by 660
  12. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
38 more
  1. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  2. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 531
  3. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  4. 2019Video Object Segmentation using Space-Time Memory Networkscited by 436
  5. 2019Video Instance Segmentationcited by 391
  6. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  7. 2018Fast Online Object Tracking and Segmentation: A Unifying Approachcited by 272
  8. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  9. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  10. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  11. 2016One-Shot Video Object Segmentationcited by 233
  12. 2019MOTS: Multi-Object Tracking and Segmentationcited by 222
  13. 2014The Secrets of Salient Object Segmentationcited by 200
  14. 2016Deep Feature Flow for Video Recognitioncited by 186
  15. 2015Visual Saliency Based on Multiscale Deep Featurescited by 185
  16. 2019FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentationcited by 178
  17. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  18. 2017Playing for Benchmarkscited by 169
  19. 2015Conditional Computation in Neural Networks for faster modelscited by 166
  20. 2018Tracking Emerges by Colorizing Videoscited by 166
  21. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  22. 2016Deep Interactive Object Selectioncited by 160
  23. 2021Track to Detect and Segment: An Online Multi-Object Trackercited by 156
  24. 2016Learning Video Object Segmentation from Static Imagescited by 151
  25. 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
  26. 2020Collaborative Video Object Segmentation by Foreground-Background Integrationcited by 142
  27. 2020Video Panoptic Segmentationcited by 141
  28. 2020See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networkscited by 138
  29. 2018Improving Semantic Segmentation via Video Propagation and Label Relaxationcited by 134
  30. 2021Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentationcited by 133
  31. 2021Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusioncited by 129
  32. 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
  33. 2020STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videoscited by 121
  34. 2020Video Object Segmentation with Episodic Graph Memory Networkscited by 118
  35. 2020Kernelized Memory Network for Video Object Segmentationcited by 111
  36. 2017Online Adaptation of Convolutional Neural Networks for Video Object Segmentationcited by 109
  37. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  38. 2020SipMask: Spatial Information Preservation for Fast Image and Video Instance Segmentationcited by 108

50 most cited of 144 All 144 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 24; 2023: 44; 2024: 27; 2025: 28; 2026: 7.

132 papers.

Sort:

By area:

  1. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 232
  2. 2023Segment and Track Anythingcited by 193
  3. 2022Decoupling Features in Hierarchical Propagation for Video Object Segmentationcited by 146
  4. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  5. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  6. 2022A Generalist Framework for Panoptic Segmentation of Images and Videoscited by 99
  7. 2022MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Trainingcited by 94
  8. 2023Referring Multi-Object Trackingcited by 91
  9. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  10. 2022Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentationcited by 64
  11. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 59
  12. 2023XMem++: Production-level Video Segmentation From Few Annotated Framescited by 53
38 more
  1. 2023CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentationcited by 52
  2. 2023Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimationcited by 46
  3. 2022Local-Global Context Aware Transformer for Language-Guided Video Segmentationcited by 41
  4. 2023CORE: Cooperative Reconstruction for Multi-Agent Perceptioncited by 41
  5. 2023Towards Vehicle-to-everything Autonomous Driving: A Survey on Collaborative Perceptioncited by 38
  6. 2023VideoCutLER: Surprisingly Simple Unsupervised Video Instance Segmentationcited by 36
  7. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  8. 2023OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fieldscited by 34
  9. 2022EM-driven unsupervised learning for efficient motion segmentationcited by 33
  10. 2022Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learningcited by 26
  11. 2022A Deeper Dive Into What Deep Spatiotemporal Networks Encode: Quantifying Static vs. Dynamic Informationcited by 24
  12. 2024Object Segmentation-Assisted Inter Prediction for Versatile Video Codingcited by 22
  13. 2023Boosting Video Object Segmentation via Space-time Correspondence Learningcited by 21
  14. 2024VideoMAC: Video Masked Autoencoders Meet ConvNetscited by 20
  15. 2022Scalable Video Object Segmentation with Identification Mechanismcited by 18
  16. 2023Deep Learning Technique for Human Parsing: A Survey and Outlookcited by 18
  17. 2023PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentationcited by 18
  18. 2023A spatio-temporal network for video semantic segmentation in surgical videoscited by 17
  19. 2025Vid-SME: Membership Inference Attacks against Large Video Understanding Modelscited by 17
  20. 2022Generative Adversarial Network for Future Hand Segmentation from Egocentric Videocited by 16
  21. 2022Visual Semantic Segmentation Based on Few/Zero-Shot Learning: An Overviewcited by 16
  22. 2023MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillationcited by 16
  23. 2023Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentationcited by 16
  24. 2024Visual Knowledge in the Big Model Era: Retrospect and Prospectcited by 16
  25. 2024Customize Segment Anything Model for Multi-Modal Semantic Segmentation with Mixture of LoRA Expertscited by 15
  26. 2025ROS-SAM: High-Quality Interactive Segmentation for Remote Sensing Moving Objectcited by 15
  27. 2023UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Modelcited by 14
  28. 2023Computation-efficient Deep Learning for Computer Vision: A Surveycited by 14
  29. 2025SAM2 for Image and Video Segmentation: A Comprehensive Surveycited by 13
  30. 2022RankSeg: Adaptive Pixel Classification with Image Category Ranking for Segmentationcited by 11
  31. 2023Spatio-Temporal Pixel-Level Contrastive Learning-based Source-Free Domain Adaptation for Video Semantic Segmentationcited by 10
  32. 2025Multimodal Large Models Are Effective Action Anticipatorscited by 10
  33. 2023Event-Free Moving Object Segmentation from Moving Ego Vehiclecited by 8
  34. 2025SAM-Body4D: Training-Free 4D Human Body Mesh Recovery from Videoscited by 8
  35. 2022In-N-Out Generative Learning for Dense Unsupervised Video Segmentationcited by 7
  36. 2022Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networkscited by 7
  37. 2024X-Prompt: Multi-modal Visual Prompt for Video Object Segmentationcited by 7
  38. 2025EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real Worldcited by 7

50 most cited of 132 All 132 (JSON)