# DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation

arXiv: https://arxiv.org/abs/2609.23103 · HTML: https://mlsanity.com/paper/2609.23103/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 32 references that are in the map.
- First posted on arXiv 19 Sep 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 27 of 27)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos](https://mlsanity.com/paper/2503.17973/) (2025)
- [Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation](https://mlsanity.com/paper/2505.02836/) (2025)
- [Gen2Sim: Scaling up Robot Learning in Simulation with Generative Models](https://mlsanity.com/paper/2310.18308/) (2023)
- [OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion](https://mlsanity.com/paper/2507.06165/) (2025)
- [SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent](https://mlsanity.com/paper/2509.20414/) (2025)
- [SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes](https://mlsanity.com/paper/2602.09153/) (2026)
- [Articraft: An Agentic System for Scalable Articulated 3D Asset Generation](https://mlsanity.com/paper/2605.15187/) (2026)
- [DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness](https://mlsanity.com/paper/2503.22677/) (2025)
- [URDF-Anything: Constructing Articulated Objects with 3D Multimodal Language Model](https://mlsanity.com/paper/2511.00940/) (2025)
- [SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning](https://mlsanity.com/paper/2605.09423/) (2026)
- [GRS: Generating Robotic Simulation Tasks from Real-World Images](https://mlsanity.com/paper/2410.15536/) (2024)
- [MVImgNet2.0: A Larger-scale Dataset of Multi-view Images](https://mlsanity.com/paper/2412.01430/) (2024)
- [ArtGS:3D Gaussian Splatting for Interactive Visual-Physical Modeling and Manipulation of Articulated Objects](https://mlsanity.com/paper/2507.02600/) (2025)
- [PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects](https://mlsanity.com/paper/2605.21572/) (2026)
- [EMPM: Embodied MPM for Modeling and Simulation of Deformable Objects](https://mlsanity.com/paper/2601.17251/) (2026)
- [PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World](https://mlsanity.com/paper/2605.05163/) (2026)
- [Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents](https://mlsanity.com/paper/2607.19190/) (2026)
- [PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement](https://mlsanity.com/paper/2602.14968/) (2026)
- [SOPHY: Learning to Generate Simulation-Ready Objects with Physical Materials](https://mlsanity.com/paper/2504.12684/) (2025)
- [PokeFlex: A Real-World Dataset of Volumetric Deformable Objects for Robotics](https://mlsanity.com/paper/2410.07688/) (2024)
- [A Convex Formulation of Frictional Contact between Rigid and Deformable Bodies](https://mlsanity.com/paper/2303.08912/) (2023)
- [SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds](https://mlsanity.com/paper/2604.08544/) (2026)

## Built on by (most cited first, 1 of 1)

- [EmbodiedSmith: Scaling Embodied Data through Recursive Self-Improvement Flywheel in Simulation](https://mlsanity.com/paper/2610.07969/) (2026)

## Abstract (arXiv)

While simulation-ready deformable assets are essential for in-silico robotic manipulation tasks, existing generation frameworks typically assess physical plausibility after generation, leaving an object's simulated response unused as feedback for repairing upstream errors. We present DiagGen, an agentic framework that turns a single in-the-wild image into a simulation-ready deformable asset through a generate--simulate--diagnose--refine loop. DiagGen constructs part-aware geometry and material parameters, then uses a VLM (vision-language model)-based agent to select semantically informative regions, probe them in a physics simulator, observe material responses, and route evidence-backed repair cues to the responsible generation stage. Experiments on 40 assets show that diagnostics provides useful repair cues and can moderately improve the quality of generated deformable assets. Finally, we show that unlike assets generated from visual foundation models which may not be simulatable, DiagGen-generated deformables can be directly dropped into a high-fidelity physical simulator for the planning and simulation of contact-rich pick-and-place tasks. The project's website is https://diaggen.github.io/.