mlsanityData to 7 Oct 2026

On Moving Object Segmentation from Monocular Video with Transformers

arXiv 2411.19141 ↗First posted 28 Nov 2024cs.CV cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Moving object detection and segmentation from a single moving camera is a challenging task, requiring an understanding of recognition, motion and 3D geometry. Combining both recognition and reconstruction boils down to a fusion problem, where appearance and motion features need to be combined for classification and segmentation. In this paper, we present a novel fusion architecture for monocular motion segmentation - M3Former, which leverages the strong performance of transformers for segmentation and multi-modal fusion. As reconstructing motion from monocular video is ill-posed, we systematically analyze different 2D and 3D motion representations for this problem and their importance for segmentation performance. Finally, we analyze the effect of training data and show that diverse datasets are required to achieve SotA performance on Kitti and Davis.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 56 of its 87 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

56 of 87 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2023Segment Anythingcited by 8,158
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2017Mask R-CNNcited by 7,259
  8. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  9. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  10. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  11. 2021Vision Transformers for Dense Predictioncited by 1,789
  12. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
38 more
  1. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  2. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  3. 2015A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimationcited by 878
  4. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 673
  5. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  6. 2019Video Instance Segmentationcited by 391
  7. 2018YouTube-VOS: A Large-Scale Video Object Segmentation Benchmarkcited by 381
  8. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  9. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  10. 2018Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentationcited by 222
  11. 2023Robust Dynamic Radiance Fieldscited by 193
  12. 2020Robust Consistent Video Depth Estimationcited by 188
  13. 2021Mask2Former for Video Instance Segmentationcited by 169
  14. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 165
  15. 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
  16. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  17. 2020See More, Know More: Unsupervised Video Object Segmentation with Co-Attention Siamese Networkscited by 138
  18. 2019Neural-Guided RANSAC: Learning Where to Sample Model Hypothesescited by 129
  19. 2020RAFT-3D: Scene Flow using Rigid-Motion Embeddingscited by 121
  20. 2022VITA: Video Instance Segmentation via Object Token Associationcited by 94
  21. 2022ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wildcited by 90
  22. 2017SfM-Net: Learning of Structure and Motion from Videocited by 89
  23. 2017FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videoscited by 77
  24. 2019Unsupervised Moving Object Detection via Contextual Information Separationcited by 72
  25. 2016It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videoscited by 71
  26. 2017Learning Video Object Segmentation with Visual Memorycited by 66
  27. 2022Segmenting Moving Objects via an Object-Centric Layered Representationcited by 63
  28. 2020Betrayed by Motion: Camouflaged Object Discovery via Motion Segmentationcited by 62
  29. 2021Consistent Depth of Moving Objects in Videocited by 56
  30. 2019Towards Segmenting Anything That Movescited by 51
  31. 2021Learning to Segment Rigid Motions from Two Framescited by 45
  32. 2021The Emergence of Objectness: Learning Zero-Shot Segmentation from Videoscited by 45
  33. 2022Discovering Objects that Can Movecited by 44
  34. 2022Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motioncited by 42
  35. 2018Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimationcited by 38
  36. 2018Object Discovery in Videos as Foreground Motion Clusteringcited by 32
  37. 2023Prismer: A Vision-Language Model with Multi-Task Expertscited by 28
  38. 2023Object Discovery from Motion-Guided Tokenscited by 25

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025Segment Any Motion in Videoscited by 34
  2. 2025Instance-Level Moving Object Segmentation from a Single Image with Eventscited by 6
  3. 2026WildRayZer: Self-supervised Large View Synthesis in Dynamic Environmentscited by 4
  4. 2025Channel-wise Motion Features for Efficient Motion Segmentationcited by 1
  5. 2025Moving Object Detection from Moving Camera Using Focus of Expansion Likelihood and Segmentationcited by 0
  6. 2026STAR-VLM: Spatiotemporal Grounding Vision-Language Models for Motion and Velocity Estimation via Automotive Radar Supervisioncited by 0