mlsanityData to 14 Oct 2026

NeuralDiff: Segmenting 3D objects that move in egocentric videos

arXiv 2110.09936 ↗First posted 19 Oct 2021cs.CV cs.GRcited by 57

Not on the map. Why?

Abstract (arXiv)

Given a raw video sequence taken from a freely-moving camera, we study the problem of decomposing the observed 3D scene into a static background and a dynamic foreground containing the objects that move in the video sequence. This task is reminiscent of the classic background subtraction problem, but is significantly harder because all parts of the scene, static and dynamic, generate a large apparent motion due to the camera large viewpoint change. In particular, we consider egocentric videos and further separate the dynamic component into objects and the actor that observes and moves them. We achieve this factorization by reconstructing the video via a triple-stream neural rendering network that explains the different motions based on corresponding inductive biases. We demonstrate that our method can successfully separate the different types of motion, outperforming recent neural rendering baselines at this task, and can accurately segment moving objects. We do so by assessing the method empirically on challenging videos from the EPIC-KITCHENS dataset which we augment with appropriate annotations to create a new benchmark for the task of dynamic object segmentation on unconstrained video sequences, for complex 3D environments.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 19 Oct 2021.

Builds on

24 of 44 references.

References · 24

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  4. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  5. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  6. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  7. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 861
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  9. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
  10. 2020Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Videocited by 471
  11. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  12. 2019The VIA Annotation Software for Images, Audio and Videocited by 167
12 more
  1. 2021Self-supervised Video Object Segmentation by Motion Groupingcited by 147
  2. 2021Animatable Neural Radiance Fields from Monocular RGB Videoscited by 109
  3. 2021Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentationcited by 88
  4. 2020Zero-Shot Video Object Segmentation via Attentive Graph Neural Networkscited by 84
  5. 2017FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videoscited by 77
  6. 2020STaR: Self-supervised Tracking and Reconstruction of Rigid Objects in Motion with Neural Renderingcited by 76
  7. 2016It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videoscited by 71
  8. 2016Learning Motion Patterns in Videoscited by 53
  9. 2021FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modellingcited by 47
  10. 2021Learning to Segment Rigid Motions from Two Framescited by 46
  11. 2018Object Discovery in Videos as Foreground Motion Clusteringcited by 32
  12. 2017Learning 3D Object Categories by Looking Around Themcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 18; 2024: 18; 2025: 8; 2026: 4.

57 papers.

Sort:

By area:

  1. 2021Neural Fields in Visual Computing and Beyondcited by 540
  2. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 371
  3. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 196
  4. 2022NeRFPlayer: A Streamable Dynamic Scene Representation with Decomposed Neural Radiance Fieldscited by 152
  5. 2022D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Videocited by 137
  6. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 135
  7. 2023Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decompositioncited by 134
  8. 2023RobustNeRF: Ignoring Distractors with Robust Lossescited by 101
  9. 2023GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View Synthesiscited by 95
  10. 2024NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wildcited by 85
  11. 2024Fully Explicit Dynamic Gaussian Splattingcited by 76
  12. 2023Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusioncited by 64
38 more
  1. 2024LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometrycited by 56
  2. 2023EPIC Fields: Marrying 3D Geometry and Video Understandingcited by 49
  3. 2024EgoLifter: Open-world 3D Segmentation for Egocentric Perceptioncited by 47
  4. 2023ObjectSDF++: Improved Object-Compositional Neural Implicit Surfacescited by 39
  5. 2024Recent Trends in 3D Reconstruction of General Non-Rigid Scenescited by 28
  6. 2022EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queriescited by 25
  7. 2023Interactive Segment Anything NeRF with Feature Imitationcited by 24
  8. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  9. 2024EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splattingcited by 23
  10. 2023Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videoscited by 21
  11. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  12. 2022Neural Groundplans: Persistent Neural Scene Representations from a Single Imagecited by 17
  13. 2024NeRFs in Robotics: A Surveycited by 16
  14. 2025DeGauss: Dynamic-Static Decomposition with Gaussian Splatting for Distractor-free 3D Reconstructioncited by 16
  15. 2024RoDUS: Robust Decomposition of Static and Dynamic Elements in Urban Scenescited by 14
  16. 2023Semantic Attention Flow Fields for Monocular Dynamic Scene Decompositioncited by 13
  17. 2023MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Fieldcited by 12
  18. 20233D Reconstruction of Objects in Hands without Real World 3D Supervisioncited by 12
  19. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  20. 2022Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Mapscited by 7
  21. 2023Replay: Multi-modal Multi-view Acted Videos for Casual Holographycited by 7
  22. 2022ONeRF: Unsupervised 3D Object Segmentation from Multiple Viewscited by 6
  23. 2024CrossScore: Towards Multi-View Image Evaluation and Scoringcited by 6
  24. 2024Planar Gaussian Splattingcited by 6
  25. 2024ClusteringSDF: Self-Organized Neural Implicit Surfaces for 3D Decompositioncited by 5
  26. 2024OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videoscited by 5
  27. 2023Factored Neural Representation for Scene Understandingcited by 4
  28. 2023Aria-NeRF: Multimodal Egocentric View Synthesiscited by 4
  29. 2024Dynamic Scene Understanding through Object-Centric Voxelization and Neural Renderingcited by 4
  30. 2025EgoSplat: Open-Vocabulary Egocentric Scene Understanding with Language Embedded 3D Gaussian Splattingcited by 4
  31. 2023DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelizationcited by 3
  32. 2025SDD-4DGS: Static-Dynamic Aware Decoupling in Gaussian Splatting for 4D Scene Reconstructioncited by 3
  33. 2025EAGLE: Episodic Appearance- and Geometry-aware Memory for Unified 2D-3D Visual Query Localization in Egocentric Visioncited by 3
  34. 2023MARF: The Medial Atom Ray Field Object Representationcited by 2
  35. 2024PruNeRF: Segment-Centric Dataset Pruning via 3D Spatial Consistencycited by 2
  36. 2024ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimizationcited by 1
  37. 2026DP-DeGauss: Dynamic Probabilistic Gaussian Decomposition for Egocentric 4D Scene Reconstructioncited by 1
  38. 2024M2DNeRF: Multi-Modal Decomposition NeRF with 3D Feature Fieldscited by 0

50 most cited of 57 All 57 (JSON)