mlsanityData to 14 Oct 2026

RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completion

arXiv 2506.05285 ↗First posted 5 Jun 2025cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

3D shape completion has broad applications in robotics, digital twin reconstruction, and extended reality (XR). Although recent advances in 3D object and scene completion have achieved impressive results, existing methods lack 3D consistency, are computationally expensive, and struggle to capture sharp object boundaries. Our work (RaySt3R) addresses these limitations by recasting 3D shape completion as a novel view synthesis problem. Specifically, given a single RGB-D image and a novel viewpoint (encoded as a collection of query rays), we train a feedforward transformer to predict depth maps, object masks, and per-pixel confidence scores for those query rays. RaySt3R fuses these predictions across multiple query views to reconstruct complete 3D shapes. We evaluate RaySt3R on synthetic and real-world datasets, and observe it achieves state-of-the-art performance, outperforming the baselines on all datasets by up to 44% in 3D chamfer distance. Project page: https://rayst3r.github.io

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 49 of its 56 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

49 of 56 references.

References · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  7. 2022Video Diffusion Modelscited by 2,305
  8. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  9. 2021Vision Transformers for Dense Predictioncited by 1,803
  10. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  11. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  12. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
37 more
  1. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  2. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  3. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
  4. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  5. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  6. 2020Convolutional Occupancy Networkscited by 761
  7. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 612
  8. 2022LION: Latent Point Diffusion Models for 3D Shape Generationcited by 553
  9. 2025Continuous 3D Perception Model with Persistent Statecited by 548
  10. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 432
  11. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 429
  12. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
  13. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  14. 2023VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completioncited by 313
  15. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 189
  16. 2024MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motioncited by 162
  17. 2024Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Imagecited by 135
  18. 2020Anisotropic Convolutional Networks for 3D Semantic Scene Completioncited by 127
  19. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  20. 2019HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objectscited by 110
  21. 2021TransformerFusion: Monocular RGB Scene Reconstruction using Transformerscited by 106
  22. 2022ShapeFormer: Transformer-based Shape Completion via Sparse Representationcited by 106
  23. 2025CAST: Component-Aligned 3D Scene Reconstruction from an RGB Imagecited by 106
  24. 2022POCO: Point Convolution for Surface Reconstructioncited by 92
  25. 2020se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domainscited by 90
  26. 20226-DoF Pose Estimation of Household Objects for Robotic Manipulation: An Accessible Dataset and Benchmarkcited by 86
  27. 2023PC2: Projection-Conditioned Point Cloud Diffusion for Single-Image 3D Reconstructioncited by 84
  28. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 77
  29. 2023Multiview Compressive Coding for 3D Reconstructioncited by 71
  30. 2021CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulationcited by 49
  31. 2020Deep Octree-based CNNs with Output-Guided Skip Connections for 3D Shape and Scene Completioncited by 33
  32. 2019RevealNet: Seeing Behind Objects in RGB-D Scanscited by 30
  33. 2024Local Policies Enable Zero-shot Long-horizon Manipulationcited by 22
  34. 2022RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Controlcited by 10
  35. 2025LaRI: Layered Ray Intersections for Single-view 3D Geometric Reasoningcited by 8
  36. 2023Constant-time Motion Planning with Anytime Refinement for Manipulationcited by 4
  37. 2022Mechanical Search on Shelves with Efficient Stacking and Destacking of Objectscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025MapAnything: Universal Feed-Forward Metric 3D Reconstructioncited by 314
  2. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  3. 2025Any4D: Unified Feed-Forward Metric 4D Reconstructioncited by 35
  4. 2026Points-to-3D: Structure-Aware 3D Generation with Point Cloud Priorscited by 4
  5. 2026GATOR: Generative and Agentic 3D Object Reconstruction From Casual Imagescited by 0