# OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects

arXiv: https://arxiv.org/abs/2407.08711 · HTML: https://mlsanity.com/paper/2407.08711/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 57 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 30 of 51)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence](https://mlsanity.com/paper/2305.15347/) (2023)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Probabilistic and Geometric Depth: Detecting Objects in Perspective](https://mlsanity.com/paper/2107.14160/) (2021)
- [Iterative Geometry Encoding Volume for Stereo Matching](https://mlsanity.com/paper/2303.06615/) (2023)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022)

## Built on by (most cited first, 22 of 22)

- [Cubify Anything: Scaling Indoor 3D Object Detection](https://mlsanity.com/paper/2412.04458/) (2024)
- [Open Vocabulary Monocular 3D Object Detection](https://mlsanity.com/paper/2411.16833/) (2024)
- [One2Any: One-Reference 6D Pose Estimation for Any Object](https://mlsanity.com/paper/2505.04109/) (2025)
- [Orientation Matters: Making 3D Generative Models Orientation-Aligned](https://mlsanity.com/paper/2506.08640/) (2025)
- [Scalable Unseen Objects 6-DoF Absolute Pose Estimation with Robotic Integration](https://mlsanity.com/paper/2503.05578/) (2025)
- [SceneDesigner: Controllable Multi-Object Image Generation with 9-DoF Pose Manipulation](https://mlsanity.com/paper/2511.16666/) (2025)
- [Weakly-Supervised Learning of Dense Functional Correspondences](https://mlsanity.com/paper/2509.03893/) (2025)
- [Beyond 'Templates': Category-Agnostic Object Pose, Size, and Shape Estimation from a Single View](https://mlsanity.com/paper/2510.11687/) (2025)
- [Coca-Splat: Collaborative Optimization for Camera Parameters and 3D Gaussians](https://mlsanity.com/paper/2504.00639/) (2025)
- [Mind the Shape Gap: A Benchmark and Baseline for Deformation-Aware 6D Pose Estimation of Agricultural Produce](https://mlsanity.com/paper/2603.27429/) (2026)
- [DynOPETs: A Versatile Benchmark for Dynamic Object Pose Estimation and Tracking in Moving Camera Scenarios](https://mlsanity.com/paper/2503.19625/) (2025)
- [Category-Level and Open-Set Object Pose Estimation for Robotics](https://mlsanity.com/paper/2504.19572/) (2025)
- [3DRot: Rediscovering the Missing Primitive for RGB-Based 3D Augmentation](https://mlsanity.com/paper/2508.01423/) (2025)
- [SCOPE: Semantic Conditioning for Sim2Real Category-Level Object Pose Estimation in Robotics](https://mlsanity.com/paper/2509.24572/) (2025)
- [Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks](https://mlsanity.com/paper/2511.02830/) (2025)
- [MoCA3D: Monocular 3D Bounding Box Prediction in the Image Plane](https://mlsanity.com/paper/2603.19538/) (2026)
- [Object Pose Transformer: Unifying Unseen Object Pose Estimation](https://mlsanity.com/paper/2603.23370/) (2026)
- [Semantic Prior Guided One-View 6D Pose Estimation for Novel Objects](https://mlsanity.com/paper/2605.07023/) (2026)
- [Category-Level 3D Correspondence in Camera Space via Morphable Object Priors](https://mlsanity.com/paper/2605.28257/) (2026)
- [SUFLECA: Scaling Up Feature Learning for CAD-to-image Alignment](https://mlsanity.com/paper/2607.15058/) (2026)
- [Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation](https://mlsanity.com/paper/2608.22757/) (2026)
- [LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning](https://mlsanity.com/paper/2609.36219/) (2026)

## Abstract (arXiv)

We propose OmniNOCS, a large-scale monocular dataset with 3D Normalized Object Coordinate Space (NOCS) maps, object masks, and 3D bounding box annotations for indoor and outdoor scenes. OmniNOCS has 20 times more object classes and 200 times more instances than existing NOCS datasets (NOCS-Real275, Wild6D). We use OmniNOCS to train a novel, transformer-based monocular NOCS prediction model (NOCSformer) that can predict accurate NOCS, instance masks and poses from 2D object detections across diverse classes. It is the first NOCS model that can generalize to a broad range of classes when prompted with 2D boxes. We evaluate our model on the task of 3D oriented bounding box prediction, where it achieves comparable results to state-of-the-art 3D detection methods such as Cube R-CNN. Unlike other 3D detection methods, our model also provides detailed and accurate 3D object shape and segmentation. We propose a novel benchmark for the task of NOCS prediction based on OmniNOCS, which we hope will serve as a useful baseline for future work in this area. Our dataset and code will be at the project website: https://omninocs.github.io.