# L3GO: Language Agents with Chain-of-3D-Thoughts for Generating Unconventional Objects

arXiv: https://arxiv.org/abs/2402.09052 · HTML: https://mlsanity.com/paper/2402.09052/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 2 of its 39 references that are in the map.
- First posted on arXiv 14 Feb 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 2 of 2)

- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)

## Built on by (most cited first, 28 of 28)

- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs](https://mlsanity.com/paper/2404.15676/) (2024)
- [Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models](https://mlsanity.com/paper/2411.16602/) (2024)
- [The Scene Language: Representing Scenes with Programs, Words, and Embeddings](https://mlsanity.com/paper/2410.16770/) (2024)
- [BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing](https://mlsanity.com/paper/2504.01786/) (2025)
- [BlenderAlchemy: Editing 3D Graphics with Vision-Language Models](https://mlsanity.com/paper/2404.17672/) (2024)
- [LL3M: Large Language 3D Modelers](https://mlsanity.com/paper/2508.08228/) (2025)
- [IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering](https://mlsanity.com/paper/2506.23329/) (2025)
- [Kubrick: Multimodal Agent Collaborations for Synthetic Video Generation](https://mlsanity.com/paper/2408.10453/) (2024)
- [A Survey On Text-to-3D Contents Generation In The Wild](https://mlsanity.com/paper/2405.09431/) (2024)
- [Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Models](https://mlsanity.com/paper/2408.11801/) (2024)
- [VLMaterial: Procedural Material Generation with Large Vision-Language Models](https://mlsanity.com/paper/2501.18623/) (2025)
- [DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation](https://mlsanity.com/paper/2412.15200/) (2024)
- [ParSEL: Parameterized Shape Editing with Language](https://mlsanity.com/paper/2405.20319/) (2024)
- [MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs](https://mlsanity.com/paper/2607.08970/) (2026)
- [MetaGen: A DSL, Database, and Benchmark for VLM-Assisted Metamaterial Generation](https://mlsanity.com/paper/2508.17568/) (2025)
- [ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling](https://mlsanity.com/paper/2510.17603/) (2025)
- [Nova3D: Code-Native Generation of Programmable 3D Assets](https://mlsanity.com/paper/2607.22738/) (2026)
- [HOTS3D: Hyper-Spherical Optimal Transport for Semantic Alignment of Text-to-3D Generation](https://mlsanity.com/paper/2407.14419/) (2024)
- [UniTransfer: Video Concept Transfer via Progressive Spatial and Timestep Decomposition](https://mlsanity.com/paper/2509.21086/) (2025)
- [Proc3D: Procedural 3D Generation and Parametric Editing of 3D Shapes with Large Language Models](https://mlsanity.com/paper/2601.12234/) (2026)
- [Reflecting Process Expertise in Procedural Material Generation](https://mlsanity.com/paper/2607.13318/) (2026)
- [ViSculpt: Visual-Centric Agentic Geometry Editing](https://mlsanity.com/paper/2608.24169/) (2026)
- [TuneGenie: Reasoning-based LLM agents for preferential music generation](https://mlsanity.com/paper/2506.12083/) (2025)
- [See it. Say it. Sorted: Agentic System for Compositional Diagram Generation](https://mlsanity.com/paper/2508.15222/) (2025)
- [ViewStruct: View-Grounded Articulation and Connectivity Inference](https://mlsanity.com/paper/2610.16793/) (2026)
- [PhysCodeGym: Benchmarking Coding Agents for Physically Verifiable 4D Reconstruction](https://mlsanity.com/paper/2610.17288/) (2026)

## Abstract (arXiv)

Diffusion-based image generation models such as DALL-E 3 and Stable Diffusion-XL demonstrate remarkable capabilities in generating images with realistic and unique compositions. Yet, these models are not robust in precisely reasoning about physical and spatial configurations of objects, especially when instructed with unconventional, thereby out-of-distribution descriptions, such as "a chair with five legs". In this paper, we propose a language agent with chain-of-3D-thoughts (L3GO), an inference-time approach that can reason about part-based 3D mesh generation of unconventional objects that current data-driven diffusion models struggle with. More concretely, we use large language models as agents to compose a desired object via trial-and-error within the 3D simulation environment. To facilitate our investigation, we develop a new benchmark, Unconventionally Feasible Objects (UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender where language agents can build and compose atomic building blocks via API calls. Human and automatic GPT-4V evaluations show that our approach surpasses the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our approach outperforms other state-of-the-art text-to-2D image and text-to-3D models based on human evaluation.