# Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D arXiv: https://arxiv.org/abs/2604.05212 ยท HTML: https://mlsanity.com/paper/2604.05212/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 48 of its 54 references that are in the map. - First posted on arXiv 6 Apr 2026. ## Builds on (most cited first, 30 of 48) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025) - [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017) - [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025) - [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019) - [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020) - [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025) - [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020) - [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021) - [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022) - [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020) - [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022) - [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022) - [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021) - [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023) - [Scaling Open-Vocabulary Object Detection](https://mlsanity.com/paper/2306.09683/) (2023) - [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025) - [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023) - [Scalable 3D Captioning with Pretrained Models](https://mlsanity.com/paper/2306.07279/) (2023) - [Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception](https://mlsanity.com/paper/2306.06362/) (2023) - [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022) - [ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection](https://mlsanity.com/paper/2106.01178/) (2021) - [Geometry Uncertainty Projection Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2107.13774/) (2021) - [Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations](https://mlsanity.com/paper/2012.09988/) (2020) - [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018) - [H3DNet: 3D Object Detection Using Hybrid Geometric Primitives](https://mlsanity.com/paper/2006.05682/) (2020) - [Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild](https://mlsanity.com/paper/2406.09905/) (2024) ## Built on by (most cited first, 7 of 7) - [FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute](https://mlsanity.com/paper/2609.08848/) (2026) - [Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling](https://mlsanity.com/paper/2608.30821/) (2026) - [WorldSculpt: Generating Compositional Worlds from Grounded Videos](https://mlsanity.com/paper/2609.05416/) (2026) - [EgoTrack3D: A Modular Framework for Egocentric 3D Object Tracking](https://mlsanity.com/paper/2608.08016/) (2026) - [Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene Reconstruction](https://mlsanity.com/paper/2609.23796/) (2026) - [DistScene: Object-to-Scene Distillation for 3D Scene Generation](https://mlsanity.com/paper/2610.06960/) (2026) - [Fly-by-Code: Embodied Coding Agents for Aerial Manipulation with Active Visual and Physical Feedback](https://mlsanity.com/paper/2610.12549/) (2026) ## Abstract (arXiv) Detecting and localizing objects in space is a fundamental computer vision problem. While much progress has been made to solve 2D object detection, 3D object localization is much less explored and far from solved, especially for open-world categories. To address this research challenge, we propose Boxer, an algorithm to estimate static 3D bounding boxes (3DBBs) from 2D open-vocabulary object detections, posed images and optional depth either represented as a sparse point cloud or dense depth. At its core is BoxerNet, a transformer-based network which lifts 2D bounding box (2DBB) proposals into 3D, followed by multi-view fusion and geometric filtering to produce globally consistent de-duplicated 3DBBs in metric world space. Boxer leverages the power of existing 2DBB detection algorithms (e.g. DETIC, OWLv2, SAM3) to localize objects in 2D. This allows the main BoxerNet model to focus on lifting to 3D rather than detecting, ultimately reducing the demand for costly annotated 3DBB training data. Extending the CuTR formulation, we incorporate an aleatoric uncertainty for robust regression, a median depth patch encoding to support sparse depth inputs, and large-scale training with over 1.2 million unique 3DBBs. BoxerNet outperforms state-of-the-art baselines in open-world 3DBB lifting, including CuTR in egocentric settings without dense depth (0.532 vs. 0.010 mAP) and on CA-1M with dense depth available (0.412 vs. 0.250 mAP).