# Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description arXiv: https://arxiv.org/abs/2412.01398 ยท HTML: https://mlsanity.com/paper/2412.01398/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Builds on 65 of its 92 references that are in the map. - First posted on arXiv 2 Dec 2024. ## Builds on (most cited first, 30 of 65) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017) - [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019) - [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023) - [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017) - [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020) - [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021) - [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018) - [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021) - [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024) - [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022) - [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017) - [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022) - [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023) - [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023) - [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019) - [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023) - [Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts](https://mlsanity.com/paper/1406.2031/) (2014) - [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022) - [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019) - [PLA: Language-Driven Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2211.16312/) (2022) - [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023) - [Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation](https://mlsanity.com/paper/2306.11290/) (2023) - [Ditto: Building Digital Twins of Articulated Objects from Interaction](https://mlsanity.com/paper/2202.08227/) (2022) - [EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI](https://mlsanity.com/paper/2312.16170/) (2023) - [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024) - [Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation](https://mlsanity.com/paper/2406.11824/) (2024) - [SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding](https://mlsanity.com/paper/2401.09340/) (2024) ## Built on by (most cited first, 9 of 9) - [ViSTA-SLAM: Visual SLAM with Symmetric Two-view Association](https://mlsanity.com/paper/2509.01584/) (2025) - [EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templates](https://mlsanity.com/paper/2604.11038/) (2026) - [NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation](https://mlsanity.com/paper/2608.24212/) (2026) - [From Scan to Action: Leveraging Realistic Scans for Embodied Scene Understanding](https://mlsanity.com/paper/2507.17585/) (2025) - [UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation](https://mlsanity.com/paper/2603.23478/) (2026) - [Incremental Semantics-Aided Meshing from LiDAR-Inertial Odometry and RGB Direct Label Transfer](https://mlsanity.com/paper/2604.09478/) (2026) - [FunArt: Decoding Functional Structure and Articulation from Generative 3D Latents](https://mlsanity.com/paper/2609.20673/) (2026) - [Geometric and Semantic Coupling for Interaction Understanding in 3D Scenes](https://mlsanity.com/paper/2609.25247/) (2026) - [Task-Adaptive Grounded 3D-Programmers Using 2D VLMs](https://mlsanity.com/paper/2610.02021/) (2026) ## Abstract (arXiv) 3D scene understanding is a long-standing challenge in computer vision and a key component in enabling mixed reality, wearable computing, and embodied AI. Providing a solution to these applications requires a multifaceted approach that covers scene-centric, object-centric, as well as interaction-centric capabilities. While there exist numerous datasets and algorithms approaching the former two problems, the task of understanding interactable and articulated objects is underrepresented and only partly covered in the research field. In this work, we address this shortcoming by introducing: (1) Articulate3D, an expertly curated 3D dataset featuring high-quality manual annotations on 280 indoor scenes. Articulate3D provides 8 types of annotations for articulated objects, covering parts and detailed motion information, all stored in a standardized scene representation format designed for scalable 3D content creation, exchange and seamless integration into simulation environments. (2) USDNet, a novel unified framework capable of simultaneously predicting part segmentation along with a full specification of motion attributes for articulated objects. We evaluate USDNet on Articulate3D as well as two existing datasets, demonstrating the advantage of our unified dense prediction approach. Furthermore, we highlight the value of Articulate3D through cross-dataset and cross-domain evaluations and showcase its applicability in downstream tasks such as scene editing through LLM prompting and robotic policy training for articulated object manipulation. We provide open access to our dataset, benchmark, and method's source code.