# Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise

arXiv: https://arxiv.org/abs/2609.36593 · HTML: https://mlsanity.com/paper/2609.36593/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023)
- [SAGE: Scalable Agentic 3D Scene Generation for Embodied AI](https://mlsanity.com/paper/2602.10116/) (2026)
- [PADL: Language-Directed Physics-Based Character Control](https://mlsanity.com/paper/2301.13868/) (2023)
- [SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes](https://mlsanity.com/paper/2602.09153/) (2026)
- [Iterative Motion Editing with Natural Language](https://mlsanity.com/paper/2312.11538/) (2023)
- [WorldExplorer: Towards Generating Fully Navigable 3D Scenes](https://mlsanity.com/paper/2506.01799/) (2025)
- [Generating Physically Stable and Buildable Brick Structures from Text](https://mlsanity.com/paper/2505.05469/) (2025)
- [PAT3D: Physics-Augmented Text-to-3D Scene Generation](https://mlsanity.com/paper/2511.21978/) (2025)
- [FactorSim: Generative Simulation via Factorized Representation](https://mlsanity.com/paper/2409.17652/) (2024)
- [Procedural Scene Programs for Open-Universe Scene Generation: LLM-Free Error Correction via Program Search](https://mlsanity.com/paper/2510.16147/) (2025)
- [Learn2Fold: Structured Origami Generation with World Model Planning](https://mlsanity.com/paper/2603.29585/) (2026)
- [GS-Agent: Creating 4D Physical Worlds With Generative Simulation](https://mlsanity.com/paper/2607.21522/) (2026)
- [CueTip: An Interactive and Explainable Physics-aware Pool Assistant](https://mlsanity.com/paper/2501.18291/) (2025)
- [ChronoLLM: Customizing Language Models for Physics-Based Simulation Code Generation](https://mlsanity.com/paper/2508.13975/) (2025)
- [Robust and Efficient Penetration-Free Elastodynamics without Barriers](https://mlsanity.com/paper/2512.12151/) (2025)
- [RelaxFlow: Text-Driven Amodal 3D Generation](https://mlsanity.com/paper/2603.05425/) (2026)

## Abstract (arXiv)

Creating diverse physical simulations remains labor-intensive because assets, layout, physical parameters, motion, control, and rendering must be designed and debugged jointly. We present Text2Sim, a simulation-specialized agentic pipeline that converts a text-only request into an executable, editable dynamic case. Built on Genesis, Text2Sim uses a hierarchical agentic structure that combines a Planner with specialized Writers, asset-generation tools, and an independent Critic. Compact skills (Debug Cards) distilled from graphics demonstrations provide role-specific physical guidance for execution-based repair. We evaluate physical quality, visual quality, and human preference on 42 held-out prompts spanning rigid, articulated, deformable, and cloth phenomena, with a paper-level split between experience construction and evaluation. We design automatic physical and visual scorers to evaluate the quality of the results, and Text2Sim achieves higher scores than all four state-of-the-art baselines on both metrics. In blinded user studies with these baselines, significantly more participants prefer Text2Sim than prefer the baselines, which is consistent with the results from our automatic scorers. The pipeline also supports a broad range of downstream applications; we select dataset construction and extension to multimodal input as two representative examples. We will release the code, the Debug Card library, and a dataset of generated cases, each pairing the text prompt and rendered video with the executable program, assets, physical parameters, controls, and recorded states.