mlsanityData to 7 Oct 2026

ImageNet3D: Towards General-Purpose Object-Level 3D Understanding

arXiv 2406.09613 ↗First posted 13 Jun 2024cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

A vision model with general-purpose object-level 3D understanding should be capable of inferring both 2D (e.g., class name and bounding box) and 3D information (e.g., 3D location and 3D viewpoint) for arbitrary rigid objects in natural images. This is a challenging task, as it involves inferring 3D information from 2D signals and most importantly, generalizing to rigid objects from unseen categories. However, existing datasets with object-level 3D annotations are often limited by the number of categories or the quality of annotations. Models developed on these datasets become specialists for certain categories or domains, and fail to generalize. In this work, we present ImageNet3D, a large dataset for general-purpose object-level 3D understanding. ImageNet3D augments 200 categories from the ImageNet dataset with 2D bounding box, 3D pose, 3D location annotations, and image captions interleaved with 3D information. With the new annotations available in ImageNet3D, we could (i) analyze the object-level 3D awareness of visual foundation models, and (ii) study and develop general-purpose models that infer both 2D and 3D information for arbitrary rigid objects in natural images, and (iii) integrate unified 3D models with large language models for 3D-related reasoning.. We consider two new tasks, probing of object-level 3D awareness and open vocabulary pose estimation, besides standard classification and pose estimation. Experimental results on ImageNet3D demonstrate the potential of our dataset in building vision models with stronger general-purpose object-level 3D understanding.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 44 of its 55 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

44 of 55 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
32 more
  1. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  2. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  3. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  4. 2023Segment Everything Everywhere All at Oncecited by 546
  5. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  6. 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
  7. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  8. 2022DeiT III: Revenge of the ViTcited by 417
  9. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  10. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  11. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  12. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 159
  13. 2020Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotationscited by 155
  14. 2020RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Drivingcited by 150
  15. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 140
  16. 2019Embodied Question Answering in Photorealistic Environments with Point Cloud Perceptioncited by 103
  17. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  18. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 90
  19. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  20. 2023Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Taskscited by 72
  21. 2023ChatPose: Chatting about 3D Human Posecited by 71
  22. 2023General Object Foundation Model for Images and Videos at Scalecited by 71
  23. 2022Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Datasetcited by 59
  24. 2021OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Imagescited by 46
  25. 2022Teaching Matters: Investigating the Role of Supervision in Vision Transformerscited by 45
  26. 2018StarMap for Category-Agnostic Keypoint and Viewpoint Estimationcited by 42
  27. 2023Open-vocabulary object 6D pose estimationcited by 29
  28. 2022Robust Category-Level 6D Pose Estimation with Coarse-to-Fine Rendering of Neural Featurescited by 23
  29. 20233D-Aware Visual Question Answering about Parts, Poses and Occlusionscited by 16
  30. 2023NOVUM: Neural Object Volumes for Robust Object Classificationcited by 11
  31. 2024OV9D: Open-Vocabulary Category-Level 9D Object Pose and Size Estimationcited by 10
  32. 2022Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 9; 2026: 12.

25 papers.

Sort:

By area:

  1. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  2. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 65
  3. 2025SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Modelscited by 37
  4. 2026Orient Anything V2: Unifying Orientation and Rotation Understandingcited by 22
  5. 2024DepthCues: Evaluating Monocular Depth Perception in Large Vision Modelscited by 18
  6. 2024Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuningcited by 13
  7. 2025Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactionscited by 13
  8. 2025Do It Yourself: Learning Semantic Correspondence from Pseudo-Labelscited by 13
  9. 2025Orientation Matters: Making 3D Generative Models Orientation-Alignedcited by 13
  10. 2025Test-Time Canonicalization by Foundation Models for Robust Perceptioncited by 5
  11. 2025View-Consistent Diffusion Representations for 3D-Consistent Video Generationcited by 3
  12. 2026LychSim: A Controllable and Interactive Simulation Framework for Vision Researchcited by 3
13 more
  1. 2025Interpretable 3D Neural Object Volumes for Robust Conceptual Reasoningcited by 2
  2. 2026AC3S: Adaptive Conditioning for 3D-Aware Synthetic Data Generationcited by 2
  3. 2024Semantic-Free Procedural 3D Shapes Are Surprisingly Good Teacherscited by 1
  4. 2025DINeMo: Learning Neural Mesh Models with no 3D Annotationscited by 0
  5. 2026PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Viewscited by 0
  6. 2026MAPS: A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Spacecited by 0
  7. 2026Every9D-21M: Large-Scale Real-World 9D Canonicalization of Everyday Objectscited by 0
  8. 2026SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Modelscited by 0
  9. 2026Emergence of a Shared Canonical Object Frame from In-the-Wild Videoscited by 0
  10. 2026Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generationcited by 0
  11. 2026Two Global Crops Suffice: Locating Semantic Emergence in DINO-Style Self-Supervised Learningcited by 0
  12. 2026Toward Comprehensive 3D Grounding: Orientation Grounding through Vision-Language Modelscited by 0
  13. 2026LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
  2. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight
  3. 2026A Cookbook of 3D Vision: Data, Learning Paradigms, and Application
  4. 2026Bridging the Dimensionality Gap: A Taxonomy and Survey of 2D Vision Model Adaptation for 3D Analysis
  5. 2025Towards 3D Objectness Learning in an Open World
  6. 2025Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding
  7. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding
  8. 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering