mlsanityData to 13 Oct 2026

Real2Code: Reconstruct Articulated Objects via Code Generation

arXiv 2406.08474 ↗First posted 12 Jun 2024cs.CV cs.AI cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

We present Real2Code, a novel approach to reconstructing articulated objects via code generation. Given visual observations of an object, we first reconstruct its part geometry using an image segmentation model and a shape completion model. We then represent the object parts with oriented bounding boxes, which are input to a fine-tuned large language model (LLM) to predict joint articulation as code. By leveraging pre-trained vision and language models, our approach scales elegantly with the number of articulated parts, and generalizes from synthetic training data to real world objects in unstructured environments. Experimental results demonstrate that Real2Code significantly outperforms previous state-of-the-art in reconstruction accuracy, and is the first approach to extrapolate beyond objects' structural complexity in the training set, and reconstructs objects with up to 10 articulated parts. When incorporated with a stereo reconstruction model, Real2Code also generalizes to real world objects from a handful of multi-view RGB images, without the need for depth or camera information.

Key facts. Cited by 66 papers in the map (data through 13 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

40 of 62 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  3. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,389
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  5. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,475
  6. 2020Convolutional Occupancy Networkscited by 760
  7. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 588
  8. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 563
  9. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 524
  10. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 369
  12. 2016Learning Shape Abstractions by Assembling Volumetric Primitivescited by 181
28 more
  1. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 181
  2. 2019Category-Level Articulated Object Pose Estimationcited by 158
  3. 2022Ditto: Building Digital Twins of Articulated Objects from Interactioncited by 153
  4. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  5. 2019Shape2Motion: Joint Analysis of Motion Parts and Attributes from 3D Shapescited by 118
  6. 2023PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objectscited by 116
  7. 2019Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Researchcited by 98
  8. 2018Deep Part Induction from Articulated Object Pairscited by 66
  9. 2023CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objectscited by 64
  10. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  11. 2022CLA-NeRF: Category-Level Articulated Neural Radiance Fieldcited by 55
  12. 2022OPD: Single-view 3D Openable Part Detectioncited by 53
  13. 2022Self-supervised Neural Articulated Shape and Appearance Modelscited by 51
  14. 2023Modular Visual Question Answering via Code Generationcited by 47
  15. 2021MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronizationcited by 46
  16. 2021Act the Part: Learning Interaction Strategies for Articulated Object Part Discoverycited by 45
  17. 2023Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perceptioncited by 40
  18. 2023PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observationscited by 40
  19. 2023OPDMulti: Openable Part Detection for Multiple Objectscited by 38
  20. 2020Visual Identification of Articulated Object Partscited by 36
  21. 2023Building Rearticulable Models for Arbitrary 3D Objects from 4D Point Cloudscited by 34
  22. 2022Category-Independent Articulated Object Tracking with Factor Graphscited by 24
  23. 2020PLAD: Learning to Infer Shape Programs with Pseudo-Labels and Approximate Distributionscited by 19
  24. 2022A Real2Sim2Real Method for Robust Object Grasping with Neural Surface Reconstructioncited by 18
  25. 2023RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimationcited by 13
  26. 2021Learning to Infer Kinematic Hierarchies for Novel Object Instancescited by 10
  27. 2023Online Estimation of Articulated Objects with Factor Graphs using Vision and Proprioceptive Sensingcited by 10
  28. 2022Learning to Infer 3D Shape Programs with Differentiable Renderercited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 4; 2025: 29; 2026: 33.

66 papers.

Sort:

By area:

  1. 2024Articulate-Anything: Automatic Modeling of Articulated Objects via a Vision-Language Foundation Modelcited by 93
  2. 2024SINGAPO: Single Image Controlled Generation of Articulated Parts in Objectscited by 67
  3. 2025ArtGS: Building Interactable Replicas of Complex Articulated Objects via Gaussian Splattingcited by 67
  4. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 52
  5. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 42
  6. 2025DRAWER: Digital Reconstruction and Articulation With Environment Realismcited by 29
  7. 2026Articraft: An Agentic System for Scalable Articulated 3D Asset Generationcited by 28
  8. 2025PartNeXt: A Next-Generation Dataset for Fine-Grained and Hierarchical 3D Part Understandingcited by 25
  9. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  10. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 21
  11. 2025iTACO: Interactable Digital Twins of Articulated Objects from Casually Captured RGBD Videoscited by 19
  12. 2025Infinite Mobility: Scalable High-Fidelity Synthesis of Articulated Objects via Procedural Generationcited by 18
38 more
  1. 2025GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objectscited by 17
  2. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 16
  3. 2025Dexterous World Modelscited by 16
  4. 2025VideoArtGS: Building Digital Twins of Articulated Objects from Monocular Videocited by 15
  5. 2025SplArt: Articulation Estimation and Part-Level Reconstruction with 3D Gaussian Splattingcited by 13
  6. 2025ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulationcited by 11
  7. 2025ART: Articulated Reconstruction Transformercited by 9
  8. 2026PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual Worldcited by 9
  9. 2025Kinematify: Open-Vocabulary Synthesis of High-DoF Articulated Objectscited by 8
  10. 2026ArtLLM: Generating Articulated Assets via 3D LLMcited by 8
  11. 2025Articulate That Object Part (ATOP): 3D Part Articulation via Text and Motion Personalizationcited by 7
  12. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
  13. 2025Guiding Diffusion-Based Articulated Object Generation by Partial Point Cloud Alignment and Physical Plausibility Constraintscited by 7
  14. 2025Robot Learning from Any Imagescited by 7
  15. 2026FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videoscited by 7
  16. 2025MonoMobility: Zero-Shot 3D Mobility Analysis from Monocular Videoscited by 6
  17. 2025REACT3D: Recovering Articulations for Interactive Physical 3D Scenescited by 6
  18. 2025SPARK: Sim-ready Part-level Articulated Reconstruction with VLM Knowledgecited by 6
  19. 2026OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policiescited by 6
  20. 2024AutoURDF: Unsupervised Robot Modeling from Point Cloud Frames Using Cluster Registrationcited by 5
  21. 2025A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewardscited by 5
  22. 2026MonoArt: Progressive Structural Reasoning for Monocular Articulated 3D Reconstructioncited by 5
  23. 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligencecited by 5
  24. 2025LodeStar: Long-horizon Dexterity via Synthetic Data Augmentation from Human Demonstrationscited by 3
  25. 2026Articulat3D: Reconstructing Articulated Digital Twins From Monocular Videos with Geometric and Motion Constraintscited by 3
  26. 2026FreeArtGS: Articulated Gaussian Splatting Under Free-moving Scenariocited by 3
  27. 2026NeuROK: Generative 4D Neural Object Kinematicscited by 3
  28. 2025UniArt: Unified 3D Representation for Generating 3D Articulated Objects with Open-Set Articulationcited by 2
  29. 2026Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Controlcited by 2
  30. 2025SPLATART: Articulated Gaussian Splatting with Estimated Object Structurecited by 1
  31. 2025ArticFlow: Generative Simulation of Articulated Mechanismscited by 1
  32. 2026ArtPro: Self-Supervised Articulated Object Reconstruction with Adaptive Integration of Mobility Proposalscited by 1
  33. 2026D-REX: Differentiable Real-to-Sim-to-Real Engine for Learning Dexterous Graspingcited by 1
  34. 2026MotionAnymesh: Physics-Grounded Articulation for Simulation-Ready Digital Twinscited by 1
  35. 2026GEAR: GEometry-motion Alternating Refinement for Articulated Object Modeling with Gaussian Splattingcited by 1
  36. 2026EgoFun3D: Modeling Interactive Objects from Egocentric Videos using Function Templatescited by 1
  37. 2026ArtMesh: Part-Aware Articulated Mesh Fields with Motion-Consistent Dynamicscited by 1
  38. 2026UniPhysGen: Unified Physical Grounding for Simulation-Ready 3D Assetscited by 1

50 most cited of 66 All 66 (JSON)