# LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs

arXiv: https://arxiv.org/abs/2506.11148 · HTML: https://mlsanity.com/paper/2506.11148/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 20 of 20)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [Diffusion-SDF: Text-to-Shape via Voxelized Diffusion](https://mlsanity.com/paper/2212.03293/) (2022)
- [LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models](https://mlsanity.com/paper/2411.09595/) (2024)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model](https://mlsanity.com/paper/2309.00810/) (2023)
- [Instant Multi-View Head Capture through Learnable Registration](https://mlsanity.com/paper/2306.07437/) (2023)
- [Physics-Informed Computer Vision: A Review and Perspectives](https://mlsanity.com/paper/2305.18035/) (2023)
- [The Importance of Directional Feedback for LLM-based Optimizers](https://mlsanity.com/paper/2405.16434/) (2024)
- [Lifting Multi-View Detection and Tracking to the Bird's Eye View](https://mlsanity.com/paper/2403.12573/) (2024)
- [Generative Physical AI in Vision: A Survey](https://mlsanity.com/paper/2501.10928/) (2025)
- [Estimating the Hallucination Rate of Generative AI](https://mlsanity.com/paper/2406.07457/) (2024)
- [DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data](https://mlsanity.com/paper/2406.04322/) (2024)
- [Detailed Facial Geometry Recovery from Multi-View Images by Learning an Implicit Function](https://mlsanity.com/paper/2201.01016/) (2022)
- [Text-to-3D Shape Generation](https://mlsanity.com/paper/2403.13289/) (2024)
- [Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering](https://mlsanity.com/paper/2405.15305/) (2024)
- [Large Language and Text-to-3D Models for Engineering Design Optimization](https://mlsanity.com/paper/2307.01230/) (2023)
- [CHROME: Clothed Human Reconstruction with Occlusion-Resilience and Multiview-Consistency from a Single Image](https://mlsanity.com/paper/2503.15671/) (2025)
- [Generative AI-based Prompt Evolution Engineering Design Optimization With Vision-Language Model](https://mlsanity.com/paper/2406.09143/) (2024)

## Built on by (most cited first, 4 of 4)

- [Enhancing Physical Plausibility in Video Generation by Reasoning the Implausibility](https://mlsanity.com/paper/2509.24702/) (2025)
- [Extended to Reality: Prompt Injection in 3D Environments](https://mlsanity.com/paper/2602.07104/) (2026)
- [Evolutionary Computation as Natural Generative AI](https://mlsanity.com/paper/2510.08590/) (2025)
- [Orchestrating Spatial Semantics via a Zone-Graph Paradigm for Intricate Indoor Scene Generation](https://mlsanity.com/paper/2605.02537/) (2026)

## Abstract (arXiv)

The emergence of generative artificial intelligence (GenAI) and large language models (LLMs) has revolutionized the landscape of digital content creation in different modalities. However, its potential use in Physical AI for engineering design, where the production of physically viable artifacts is paramount, remains vastly underexplored. The absence of physical knowledge in existing LLM-to-3D models often results in outputs detached from real-world physical constraints. To address this gap, we introduce LLM-to-Phy3D, a physically conform online 3D object generation that enables existing LLM-to-3D models to produce physically conforming 3D objects on the fly. LLM-to-Phy3D introduces a novel online black-box refinement loop that empowers large language models (LLMs) through synergistic visual and physics-based evaluations. By delivering directional feedback in an iterative refinement process, LLM-to-Phy3D actively drives the discovery of prompts that yield 3D artifacts with enhanced physical performance and greater geometric novelty relative to reference objects, marking a substantial contribution to AI-driven generative design. Systematic evaluations of LLM-to-Phy3D, supported by ablation studies in vehicle design optimization, reveal various LLM improvements gained by 4.5% to 106.7% in producing physically conform target domain 3D designs over conventional LLM-to-3D models. The encouraging results suggest the potential general use of LLM-to-Phy3D in Physical AI for scientific and engineering applications.