mlsanityData to 7 Oct 2026

RayTran: 3D pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers

arXiv 2203.13296 ↗First posted 24 Mar 2022cs.CVcited by 17

Not on the map. Why?

Abstract (arXiv)

We propose a transformer-based neural network architecture for multi-object 3D reconstruction from RGB videos. It relies on two alternative ways to represent its knowledge: as a global 3D grid of features and an array of view-specific 2D grids. We progressively exchange information between the two with a dedicated bidirectional attention mechanism. We exploit knowledge about the image formation process to significantly sparsify the attention weight matrix, making our architecture feasible on current hardware, both in terms of memory and computation. We attach a DETR-style head on top of the 3D feature grid in order to detect the objects in the scene and to predict their 3D pose and 3D shape. Compared to previous methods, our architecture is single stage, end-to-end trainable, and it can reason holistically about a scene from multiple video frames without needing a brittle tracking step. We evaluate our method on the challenging Scan2CAD dataset, where we outperform (1) recent state-of-the-art methods for 3D object pose estimation from RGB videos; and (2) a strong alternative method combining Multi-view Stereo with RGB-D CAD alignment. We plan to release our source code.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 51 of its 68 references that are in the map. First posted on arXiv 24 Mar 2022.

Builds on

51 of 68 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  7. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  8. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  9. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
38 more
  1. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  2. 2015ORB-SLAM: a Versatile and Accurate Monocular SLAM Systemcited by 983
  3. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  4. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  5. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  6. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  7. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 515
  8. 2016Direct Sparse Odometrycited by 479
  9. 2019Tracking without bells and whistlescited by 382
  10. 20163D Bounding Box Estimation Using Deep Learning and Geometrycited by 277
  11. 2019BSP-Net: Generating Compact Meshes via Binary Space Partitioningcited by 205
  12. 2016Learning a Predictable and Generative Vector Representation for Objectscited by 173
  13. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  14. 2021ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detectioncited by 157
  15. 2018Scan2CAD: Learning CAD Model Alignment in RGB-D Scanscited by 142
  16. 2018CubeSLAM: Monocular 3D Object SLAMcited by 101
  17. 2018QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAMcited by 86
  18. 2020Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Imagescited by 79
  19. 2016IM2CADcited by 78
  20. 2018Joint Monocular 3D Vehicle Detection and Trackingcited by 77
  21. 2018Holistic 3D Scene Parsing and Reconstruction from a Single RGB Imagecited by 69
  22. 2020DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusioncited by 68
  23. 2021ROCA: Robust CAD Model Retrieval and Alignment from a Single Imagecited by 66
  24. 2020Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrievecited by 65
  25. 2020FroDO: From Detections to 3D Objectscited by 51
  26. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  27. 2020CoReNet: Coherent 3D scene reconstruction from a single RGB imagecited by 45
  28. 2020SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scanscited by 44
  29. 2019End-to-End CAD Model Retrieval and 9DoF Alignment in 3D Scanscited by 40
  30. 2021Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Imagecited by 39
  31. 2020Vid2CAD: CAD Model Alignment using Multi-View Constraints from Videoscited by 33
  32. 2020From Points to Multi-Object 3D Reconstructioncited by 29
  33. 2021ODAM: Object Detection, Association, and Mapping using Posed RGB Videocited by 22
  34. 2018A Mixed Classification-Regression Framework for 3D Pose Estimation from 2D Imagescited by 19
  35. 2020Associative3D: Volumetric Reconstruction from Sparse Viewscited by 18
  36. 2020MOLTR: Multiple Object Localisation, Tracking, and Reconstruction from Monocular RGB Videoscited by 18
  37. 2021ELLIPSDF: Joint Object Pose and Shape Optimization with a Bi-level Ellipsoid and Signed Distance Function Descriptioncited by 10
  38. 2018Visual-Inertial Object Detection and Mappingcited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 4; 2025: 4; 2026: 1.

17 papers.

Sort:

By area:

  1. 2025SAM 3D: 3Dfy Anything in Imagescited by 230
  2. 2024SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Modelcited by 95
  3. 2024OmniGlue: Generalizable Feature Matching with Foundation Model Guidancecited by 55
  4. 2023SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understandingcited by 38
  5. 2024EFM3D: A Benchmark for Measuring Progress Towards 3D Egocentric Foundation Modelscited by 25
  6. 20223D Vision with Transformers: A Surveycited by 21
  7. 2025CamCtrl3D: Single-Image Scene Exploration with Precise 3D Camera Controlcited by 21
  8. 2025LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scanscited by 21
  9. 2023CAD-Estate: Large-scale CAD Model Annotation in RGB Videoscited by 14
  10. 2023Pixel-Aligned Recurrent Queries for Multi-View 3D Object Detectioncited by 14
  11. 2025Detection Based Part-level Articulated Object Reconstruction from Single RGBD Imagecited by 13
  12. 2023O2-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Modelcited by 8
5 more
  1. 2024FastCAD: Real-Time CAD Retrieval and Alignment from Scans and Videoscited by 7
  2. 2023LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Datasetcited by 5
  3. 2023TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimationcited by 2
  4. 2022Equivariant Light Field Convolution and Transformercited by 1
  5. 2026World2Minecraft: Occupancy-Driven Simulated Scenes Constructioncited by 0