mlsanityData to 7 Oct 2026

RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos

arXiv 2401.12592 ↗First posted 23 Jan 2024cs.CVcited by 113

Not on the map. Why?

Abstract (arXiv)

We introduce a new RGB-D object dataset captured in the wild called WildRGB-D. Unlike most existing real-world object-centric datasets which only come with RGB capturing, the direct capture of the depth channel allows better 3D annotations and broader downstream applications. WildRGB-D comprises large-scale category-level RGB-D object videos, which are taken using an iPhone to go around the objects in 360 degrees. It contains around 8500 recorded objects and nearly 20000 RGB-D videos across 46 common object categories. These videos are taken with diverse cluttered backgrounds with three setups to cover as many real-world scenarios as possible: (i) a single object in one video; (ii) multiple objects in one video; and (iii) an object with a static hand in one video. The dataset is annotated with object masks, real-world scale camera poses, and reconstructed aggregated point clouds from RGBD videos. We benchmark four tasks with WildRGB-D including novel view synthesis, camera pose estimation, object 6d pose estimation, and object surface reconstruction. Our experiments show that the large-scale capture of RGB-D objects provides a large potential to advance 3D object learning. Our project page is https://wildrgbd.github.io/.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Builds on 63 of its 85 references that are in the map. First posted on arXiv 23 Jan 2024.

Builds on

63 of 85 references.

References · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2023Segment Anythingcited by 8,158
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  10. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  11. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  12. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
38 more
  1. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  5. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  6. 2021Volume Rendering of Neural Implicit Surfacescited by 990
  7. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  8. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  9. 2018Open3D: A Modern Library for 3D Data Processingcited by 717
  10. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 673
  11. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  12. 2021UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstructioncited by 643
  13. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  14. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  15. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  16. 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 487
  17. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  18. 2019PointRend: Image Segmentation as Renderingcited by 439
  19. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  20. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  21. 2021NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Imagescited by 336
  22. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  23. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  24. 2020D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometrycited by 206
  25. 2021Neural Rays for Occlusion-aware Image-based Renderingcited by 192
  26. 2017DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networkscited by 173
  27. 2020Shape Prior Deformation for Categorical 6D Object Pose and Size Estimationcited by 158
  28. 2020Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotationscited by 155
  29. 2021Improving neural implicit surfaces geometry with patch warpingcited by 136
  30. 2020Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimationcited by 123
  31. 2021FS-Net: Fast Shape-based Network for Category-Level 6D Object Pose Estimation with Decoupled Rotation Mechanismcited by 121
  32. 2021DualPoseNet: Category-level 6D Object Pose and Size Estimation Using Dual Pose Network with Refined Learning of Pose Consistencycited by 103
  33. 2022RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wildcited by 102
  34. 2020Category Level Object Pose Estimation via Neural Analysis-by-Synthesiscited by 100
  35. 2022Voxurf: Voxel-based Efficient and Accurate Neural Surface Reconstructioncited by 92
  36. 2023RelPose++: Recovering 6D Poses from Sparse-view Observationscited by 87
  37. 2021Unsupervised Learning of 3D Object Categories from Videos in the Wildcited by 67
  38. 2016A Large Dataset of Object Scanscited by 64

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 47; 2026: 60.

113 papers.

Sort:

By area:

  1. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  2. 2024Grounding Image Matching in 3D with MASt3Rcited by 960
  3. 2025π3: Permutation-Equivariant Visual Geometry Learningcited by 292
  4. 2025AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Viewscited by 234
  5. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 179
  6. 2025FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Viewscited by 176
  7. 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 170
  8. 2025Streaming 4D Visual Geometry Transformercited by 147
  9. 2025MUSt3R: Multi-view Network for Stereo 3D Reconstructioncited by 139
  10. 2025SpatialTrackerV2: 3D Point Tracking Made Easycited by 84
  11. 2025Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memorycited by 82
  12. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
38 more
  1. 2025UniK3D: Universal Camera Monocular 3D Estimationcited by 60
  2. 2025WorldMirror: Universal 3D World Reconstruction with Any-Prior Promptingcited by 60
  3. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 59
  4. 2025Raw2Drive: Reinforcement Learning with Aligned World Models for End-to-End Autonomous Driving (in CARLA v2)cited by 45
  5. 2025Dens3R: A Foundation Model for 3D Geometry Predictioncited by 44
  6. 2025DepthVLA: Enhancing Vision-Language-Action Models with Depth-Aware Spatial Reasoningcited by 44
  7. 2026HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worldscited by 34
  8. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 33
  9. 2025WinT3R: Window-Based Streaming Reconstruction with Camera Token Poolcited by 32
  10. 2026LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstructioncited by 32
  11. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 30
  12. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 28
  13. 2026Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstructioncited by 25
  14. 2024Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attentioncited by 24
  15. 2026Gen3R: 3D Scene Generation Meets Feed-Forward Reconstructioncited by 24
  16. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
  17. 2024SuperGaussian: Repurposing Video Models for 3D Super Resolutioncited by 22
  18. 2025E-RayZer: Self-supervised 3D Reconstruction as Spatial Visual Pre-trainingcited by 20
  19. 2026VGGT-Omegacited by 20
  20. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 19
  21. 2025Sharp Monocular View Synthesis in Less Than a Secondcited by 19
  22. 2025SAIL-Recon: Large SfM by Augmenting Scene Regression with Localizationcited by 18
  23. 2026LongStream: Long-Sequence Streaming Autoregressive Visual Geometrycited by 17
  24. 2025AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backendcited by 16
  25. 2025LiteVGGT: Boosting Vanilla VGGT via Geometry-aware Cached Token Mergingcited by 16
  26. 2026WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memoriescited by 15
  27. 2026VGG-T3: Offline Feed-Forward 3D Reconstruction at Scalecited by 14
  28. 2024UNOPose: Unseen Object Pose Estimation with an Unposed RGB-D Reference Imagecited by 12
  29. 2025Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusioncited by 12
  30. 2026AnyView: Synthesizing Any Novel View in Dynamic Scenescited by 12
  31. 2025G-CUT3R: Guided 3D Reconstruction with Camera and Depth Prior Integrationcited by 11
  32. 2025Tinker: Diffusion's Gift to 3D--Multi-View Consistent Editing From Sparse Inputs without Per-Scene Optimizationcited by 9
  33. 2025Review of Feed-forward 3D Reconstruction: From DUSt3R to VGGTcited by 8
  34. 2025ACE-G: Improving Generalization of Scene Coordinate Regression Through Query Pre-Trainingcited by 8
  35. 20254D-VGGT: A General Foundation Model with SpatioTemporal Awareness for Dynamic Scene Geometry Estimationcited by 8
  36. 2025Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splattingcited by 8
  37. 2026Flow4R: Unifying 4D Reconstruction and Tracking with Scene Flowcited by 8
  38. 2026Speed3R: Sparse Feed-forward 3D Reconstruction Modelscited by 8

50 most cited of 113 All 113 (JSON)