Code2Worlds-2: Physically Plausible Agent Harness for Embodied World Simulation
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Recent coding large language models (LLMs) can synthesize dynamic 4D worlds from natural language, but the resulting content remains a visual artifact. It lacks physical semantics such as the collision geometry, mass, friction, inertia, and joint structure, required to enable it to simulate the physical world, and its dynamics are scripted to look plausible rather than designed to execute. What is needed is an executable world: one that a physics engine can load and step, resolving contact physically. We present Code2Worlds-2, an extension of Code2Worlds that turns 4D world generation into an embodied world simulation. We keep the original dual-stream generation pipeline intact and append two modules. First, a SimReady Scene Compiler that turns a generated visual scene into a simulator-compatible representation: it assigns collision meshes via convex decomposition, estimates physical attributes with a vision-language model, infers a lightweight support graph, instantiates the engine's contact, friction, and dynamics constraints, and applies a gravity settling pass, before exporting the scene as MuJoCo MJCF. Second, a Simulator-in-the-Loop 4D Refiner that implements an automated agent harness-a compile, simulate, critique and refine loop-that converts visual dynamics into executable rigid and articulated dynamics and evolves the physical settings rather than only the code, replacing the purely visual motion critic with a simulator critic that inspects penetration depth, collision pairs, stability, contact forces, joint limits, and robot reachability. We further propose SimCode4D, a state-based benchmark with five task templates for evaluating embodied simulation. Experiments on SimCode4D and across natural, indoor, industrial, urban, and service scenes show that Code2Worlds-2 produces physically plausible, simulator-compatible, and robot-interactable scenes.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 54 of its 70 references that are in the map. First posted on arXiv 13 Oct 2026.
Builds on
54 of 70 references.
Closest ancestors · 2
Same direction, newest first
- 2026SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes9 Feb
- 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code2 Mar
Other references · 52
- 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
- 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
- 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
- 2024Genie: Generative Interactive Environmentscited by 747
- 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
- 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
- 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
- 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
38 more
- 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
- 2023Learning Interactive Real-World Simulatorscited by 458
- 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
- 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
- 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
- 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
- 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
- 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
- 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
- 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
- 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
- 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 193
- 2025Ctrl-World: A Controllable Generative World Model for Robot Manipulationcited by 167
- 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
- 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 148
- 2023Infinite Photorealistic Worlds using Procedural Generationcited by 147
- 2024Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generationcited by 137
- 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 125
- 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 111
- 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 95
- 2024WorldDreamer: Towards General World Models for Video Generation via Predicting Masked Tokenscited by 89
- 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
- 2022Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Searchcited by 67
- 2024Physical Property Understanding from Language-Embedded Feature Fieldscited by 48
- 2024BlenderLLM: Training Large Language Models for Computer-Aided Design with Self-improvementcited by 29
- 2025PUGS: Zero-shot Physical Understanding with Gaussian Splattingcited by 24
- 2024GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMscited by 23
- 2024CityX: Controllable Procedural Content Generation for Unbounded 3D Citiescited by 22
- 2025LL3M: Large Language 3D Modelerscited by 20
- 2025InternScenes: A Large-scale Simulatable Indoor Scene Dataset with Realistic Layoutscited by 20
- 2025MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Cloudscited by 19
- 2026Code2Worlds: Empowering Coding LLMs for 4D World Generationcited by 19
- 2025PAT3D: Physics-Augmented Text-to-3D Scene Generationcited by 16
- 2026SimRecon: SimReady Compositional Scene Reconstruction from Real Videoscited by 12
- 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objectscited by 11
- 2025Empart: Interactive Convex Decomposition for Converting Meshes to Partscited by 2
- 2026Physically Guided Visual Mass Estimation from a Single RGB Imagecited by 2
- 2026VisACD: Visibility-Based GPU-Accelerated Approximate Convex Decompositioncited by 2
50 most cited of 52 All 52 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026ChronoAgentic: A Code-based Multi-Agent World Simulator for Physically Grounded Simulation Construction
- 2026WorldAgent: Verification-Guided Agentic Physical World Construction
- 2026Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents
- 2026GS-Agent: Creating 4D Physical Worlds With Generative Simulation
- 2026Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
- 20264DSynth: Controllable Procedural World Synthesis for Dynamic Embodied Simulation
- 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation
- 2026ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling