# Cubify Anything: Scaling Indoor 3D Object Detection

arXiv: https://arxiv.org/abs/2412.04458 · HTML: https://mlsanity.com/paper/2412.04458/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 20 references that are in the map.
- First posted on arXiv 5 Dec 2024.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022)
- [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://mlsanity.com/paper/2106.01178/) (2021)
- [ImVoteNet: Boosting 3D Object Detection in Point Clouds with Image Votes](https://mlsanity.com/paper/2001.10692/) (2020)
- [FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection](https://mlsanity.com/paper/2112.00322/) (2021)
- [Perspective Fields for Single Image Camera Calibration](https://mlsanity.com/paper/2212.03239/) (2022)
- [TR3D: Towards Real-Time Indoor 3D Object Detection](https://mlsanity.com/paper/2302.02858/) (2023)
- [OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects](https://mlsanity.com/paper/2407.08711/) (2024)

## Built on by (most cited first, 30 of 39)

- [RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2506.04308/) (2025)
- [Scaling Spatial Intelligence with Multimodal Foundation Models](https://mlsanity.com/paper/2511.13719/) (2025)
- [MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs](https://mlsanity.com/paper/2503.13111/) (2025)
- [TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2510.07181/) (2025)
- [SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence](https://mlsanity.com/paper/2505.17012/) (2025)
- [Robix: A Unified Model for Robot Interaction, Reasoning and Planning](https://mlsanity.com/paper/2509.01106/) (2025)
- [LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans](https://mlsanity.com/paper/2507.02861/) (2025)
- [Perspective-Invariant 3D Object Detection](https://mlsanity.com/paper/2507.17665/) (2025)
- [VGG-T3: Offline Feed-Forward 3D Reconstruction at Scale](https://mlsanity.com/paper/2602.23361/) (2026)
- [WildDet3D: Scaling Promptable 3D Detection in the Wild](https://mlsanity.com/paper/2604.08626/) (2026)
- [Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2512.13660/) (2025)
- [FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos](https://mlsanity.com/paper/2604.05621/) (2026)
- [Thinking with Spatial Code for Physical-World Video Reasoning](https://mlsanity.com/paper/2603.05591/) (2026)
- [Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D](https://mlsanity.com/paper/2604.05212/) (2026)
- [LabelAny3D: Label Any Object 3D in the Wild](https://mlsanity.com/paper/2601.01676/) (2026)
- [D'ej`a View: Looping Transformers for Multi-View 3D Reconstruction](https://mlsanity.com/paper/2605.30215/) (2026)
- [HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Task](https://mlsanity.com/paper/2603.14367/) (2026)
- [BoxFusion: Reconstruction-Free Open-Vocabulary 3D Object Detection via Real-Time Multi-View Box Fusion](https://mlsanity.com/paper/2506.15610/) (2025)
- [PixARMesh: Autoregressive Mesh-Native Single-View Scene Reconstruction](https://mlsanity.com/paper/2603.05888/) (2026)
- [NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data](https://mlsanity.com/paper/2603.18496/) (2026)
- [HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models](https://mlsanity.com/paper/2603.25411/) (2026)
- [Why Far Looks Up: Probing Spatial Representation in Vision-Language Models](https://mlsanity.com/paper/2605.30161/) (2026)
- [Grounded 3D-Aware Spatial Vision-Language Modeling](https://mlsanity.com/paper/2605.30307/) (2026)
- [A Review on Geometry and Surface Inspection in 3D Concrete Printing](https://mlsanity.com/paper/2503.07472/) (2025)
- [Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping](https://mlsanity.com/paper/2505.23756/) (2025)
- [SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them](https://mlsanity.com/paper/2607.27703/) (2026)
- [Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling](https://mlsanity.com/paper/2608.30821/) (2026)
- [WorldSculpt: Generating Compositional Worlds from Grounded Videos](https://mlsanity.com/paper/2609.05416/) (2026)
- [Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow](https://mlsanity.com/paper/2603.28548/) (2026)
- [SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale](https://mlsanity.com/paper/2606.13497/) (2026)

## Abstract (arXiv)

We consider indoor 3D object detection with respect to a single RGB(-D) frame acquired from a commodity handheld device. We seek to significantly advance the status quo with respect to both data and modeling. First, we establish that existing datasets have significant limitations to scale, accuracy, and diversity of objects. As a result, we introduce the Cubify-Anything 1M (CA-1M) dataset, which exhaustively labels over 400K 3D objects on over 1K highly accurate laser-scanned scenes with near-perfect registration to over 3.5K handheld, egocentric captures. Next, we establish Cubify Transformer (CuTR), a fully Transformer 3D object detection baseline which rather than operating in 3D on point or voxel-based representations, predicts 3D boxes directly from 2D features derived from RGB(-D) inputs. While this approach lacks any 3D inductive biases, we show that paired with CA-1M, CuTR outperforms point-based methods - accurately recalling over 62% of objects in 3D, and is significantly more capable at handling noise and uncertainty present in commodity LiDAR-derived depth maps while also providing promising RGB only performance without architecture changes. Furthermore, by pre-training on CA-1M, CuTR can outperform point-based methods on a more diverse variant of SUN RGB-D - supporting the notion that while inductive biases in 3D are useful at the smaller sizes of existing datasets, they fail to scale to the data-rich regime of CA-1M. Overall, this dataset and baseline model provide strong evidence that we are moving towards models which can effectively Cubify Anything.