mlsanityData to 7 Oct 2026

Video-kMaX: A Simple Unified Approach for Online and Near-Online Video Panoptic Segmentation

arXiv 2304.04694 ↗First posted 10 Apr 2023cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

Video Panoptic Segmentation (VPS) aims to achieve comprehensive pixel-level scene understanding by segmenting all pixels and associating objects in a video. Current solutions can be categorized into online and near-online approaches. Evolving over the time, each category has its own specialized designs, making it nontrivial to adapt models between different categories. To alleviate the discrepancy, in this work, we propose a unified approach for online and near-online VPS. The meta architecture of the proposed Video-kMaX consists of two components: within clip segmenter (for clip-level segmentation) and cross-clip associater (for association beyond clips). We propose clip-kMaX (clip k-means mask transformer) and HiLA-MB (Hierarchical Location-Aware Memory Buffer) to instantiate the segmenter and associater, respectively. Our general formulation includes the online scenario as a special case by adopting clip length of one. Without bells and whistles, Video-kMaX sets a new state-of-the-art on KITTI-STEP and VIPSeg for video panoptic segmentation, and VSPW for video semantic segmentation. Code will be made publicly available.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 52 of its 61 references that are in the map. First posted on arXiv 10 Apr 2023.

Builds on

52 of 61 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017Mask R-CNNcited by 7,259
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  10. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  11. 2017Deformable Convolutional Networkscited by 1,481
  12. 2018Panoptic Segmentationcited by 796
38 more
  1. 2019Panoptic Feature Pyramid Networkscited by 644
  2. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  3. 2019Video Instance Segmentationcited by 391
  4. 2021MOTR: End-to-End Multiple-Object Tracking with Transformercited by 379
  5. 2020TransTrack: Multiple Object Tracking with Transformercited by 370
  6. 2019Panoptic-DeepLabcited by 362
  7. 2020MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformerscited by 355
  8. 2020Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentationcited by 326
  9. 2020Conditional Convolutions for Instance Segmentationcited by 314
  10. 2019MOTS: Multi-Object Tracking and Segmentationcited by 222
  11. 2021K-Net: Towards Unified Image Segmentationcited by 211
  12. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  13. 2016Deep Feature Flow for Video Recognitioncited by 186
  14. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  15. 2021Mask2Former for Video Instance Segmentationcited by 169
  16. 2020Video Panoptic Segmentationcited by 141
  17. 2018Improving Semantic Segmentation via Video Propagation and Label Relaxationcited by 134
  18. 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
  19. 2022MeMOT: Multi-Object Tracking with Memorycited by 113
  20. 2021Video Instance Segmentation using Inter-Frame Communication Transformerscited by 105
  21. 2021SeqFormer: Sequential Transformer for Video Instance Segmentationcited by 98
  22. 2022In Defense of Online Models for Video Instance Segmentationcited by 97
  23. 2020ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentationcited by 95
  24. 2022VITA: Video Instance Segmentation via Object Token Associationcited by 94
  25. 2022MinVIS: A Minimal Video Instance Segmentation Framework without Video-based Trainingcited by 94
  26. 2020Fully Convolutional Networks for Panoptic Segmentationcited by 92
  27. 2022Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentationcited by 79
  28. 2018Accel: A Corrective Fusion Network for Efficient Semantic Segmentation on Videocited by 78
  29. 2021STEP: Segmenting and Tracking Every Pixelcited by 72
  30. 2018Attention-guided Unified Network for Panoptic Segmentationcited by 71
  31. 2017Semantic Video CNNs through Representation Warpingcited by 65
  32. 2019DeeperLab: Single-Shot Image Parsercited by 63
  33. 2022CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentationcited by 60
  34. 2022A Generalized Framework for Video Instance Segmentationcited by 52
  35. 2022Tracking Objects as Pixel-wise Distributionscited by 36
  36. 2022TubeFormer-DeepLab: Video Mask Transformercited by 35
  37. 2022Efficient Video Instance Segmentation via Tracklet Query and Proposalcited by 33
  38. 2021Learning to Associate Every Segment for Video Panoptic Segmentationcited by 25

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 11; 2025: 2; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  2. 2023DVIS: Decoupled Video Instance Segmentation Frameworkcited by 77
  3. 2023DVIS++: Improved Decoupled Framework for Universal Video Segmentationcited by 46
  4. 2024UniVS: Unified and Universal Video Segmentation with Prompts as Queriescited by 26
  5. 2024COCONut: Modernizing COCO Segmentationcited by 24
  6. 2023ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentationcited by 18
  7. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  8. 2024CAVIS: Context-Aware Video Instance Segmentationcited by 11
  9. 2026VidEoMT: Your ViT is Secretly Also a Video Segmentation Modelcited by 7
  10. 2024General and Task-Oriented Video Segmentationcited by 6
  11. 2024RMP-SAM: Towards Real-Time Multi-Purpose Segment Anythingcited by 5
  12. 2026Video Understanding: From Geometry and Semantics to Unified Modelscited by 5
12 more
  1. 2023A Simple Video Segmenter by Tracking Objects Along Axial Trajectoriescited by 3
  2. 2024Enhancing Temporal Consistency in Video Editing by Reconstructing Videos with 3D Gaussian Splattingcited by 2
  3. 2023SegmATRon: Embodied Adaptive Semantic Segmentation for Indoor Environmentcited by 1
  4. 2026Scene-Centric Unsupervised Video Panoptic Segmentationcited by 1
  5. 20233rd Place Solution for PVUW Challenge 2023: Video Panoptic Segmentationcited by 0
  6. 20242nd Place Solution for PVUW Challenge 2024: Video Panoptic Segmentationcited by 0
  7. 2024Semantic Segmentation on VSPW Dataset through Masked Video Consistencycited by 0
  8. 20241st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentationcited by 0
  9. 2024Rethinking Video Segmentation with Masked Video Consistency: Did the Model Learn as Intended?cited by 0
  10. 2025Learning Appearance and Motion Cues for Panoptic Trackingcited by 0
  11. 2025Local2Global query Alignment for Video Instance Segmentationcited by 0
  12. 2026Unified Video Dense Prediction from Disjoint Datacited by 0