# PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement arXiv: https://arxiv.org/abs/2602.14968 ยท HTML: https://mlsanity.com/paper/2602.14968/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. - Builds on 40 of its 63 references that are in the map. - First posted on arXiv 16 Feb 2026. ## Placed in - LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/) ## Builds on (most cited first, 30 of 40) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023) - [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024) - [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024) - [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020) - [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022) - [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023) - [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023) - [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024) - [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023) - [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023) - [RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation](https://mlsanity.com/paper/2311.01455/) (2023) - [ATISS: Autoregressive Transformers for Indoor Scene Synthesis](https://mlsanity.com/paper/2110.03675/) (2021) - [Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition](https://mlsanity.com/paper/2307.14535/) (2023) - [PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification](https://mlsanity.com/paper/2303.05512/) (2023) - [Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation](https://mlsanity.com/paper/2406.11824/) (2024) - [GenSim: Generating Robotic Simulation Tasks via Large Language Models](https://mlsanity.com/paper/2310.01361/) (2023) - [PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI](https://mlsanity.com/paper/2404.09465/) (2024) - [Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians](https://mlsanity.com/paper/2403.09434/) (2024) - [LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models](https://mlsanity.com/paper/2412.02193/) (2024) - [AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes](https://mlsanity.com/paper/2312.06644/) (2023) - [Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation](https://mlsanity.com/paper/2505.02836/) (2025) - [Imitating Task and Motion Planning with Visuomotor Transformers](https://mlsanity.com/paper/2305.16309/) (2023) - [PIE-NeRF: Physics-based Interactive Elastodynamics with NeRF](https://mlsanity.com/paper/2311.13099/) (2023) - [Physical Property Understanding from Language-Embedded Feature Fields](https://mlsanity.com/paper/2404.04242/) (2024) - [RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models](https://mlsanity.com/paper/2212.05993/) (2022) ## Built on by (most cited first, 7 of 7) - [Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents](https://mlsanity.com/paper/2607.19190/) (2026) - [REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image](https://mlsanity.com/paper/2605.30338/) (2026) - [3D Generation for Embodied AI and Robotic Simulation: A Survey](https://mlsanity.com/paper/2604.26509/) (2026) - [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026) - [DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation](https://mlsanity.com/paper/2609.23103/) (2026) - [IM-ENGINE: Image Editing for Embodied Data Generation](https://mlsanity.com/paper/2609.06279/) (2026) - [WorldAgent: Verification-Guided Agentic Physical World Construction](https://mlsanity.com/paper/2609.33208/) (2026) ## Abstract (arXiv) Automatically generating interactive 3D environments is crucial for scaling up robotic data collection in simulation. While prior work has primarily focused on 3D asset placement, it often overlooks the physical relationships between objects (e.g., contact, support, balance, and containment), which are essential for creating complex and realistic manipulation scenarios such as tabletop arrangements, shelf organization, or box packing. Compared to classical 3D layout generation, producing complex physical scenes introduces additional challenges: (a) higher object density and complexity (e.g., a small shelf may hold dozens of books), (b) richer supporting relationships and compact spatial layouts, and (c) the need to accurately model both spatial placement and physical properties. To address these challenges, we propose PhyScensis, an LLM agent-based framework powered by a physics engine, to produce physically plausible scene configurations with high complexity. Specifically, our framework consists of three main components: an LLM agent iteratively proposes assets with spatial and physical predicates; a solver, equipped with a physics engine, realizes these predicates into a 3D scene; and feedback from the solver informs the agent to refine and enrich the configuration. Moreover, our framework preserves strong controllability over fine-grained textual descriptions and numerical parameters (e.g., relative positions, scene stability), enabled through probabilistic programming for stability and a complementary heuristic that jointly regulates stability and spatial relations. Experimental results show that our method outperforms prior approaches in scene complexity, visual quality, and physical accuracy, offering a unified pipeline for generating complex physical scene layouts for robotic manipulation.