# Re-Thinking Inverse Graphics With Large Language Models

arXiv: https://arxiv.org/abs/2404.15228 · HTML: https://mlsanity.com/paper/2404.15228/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 4 of its 137 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 4 of 4)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [3D-GPT: Procedural 3D Modeling with Large Language Models](https://mlsanity.com/paper/2310.12945/) (2023)

## Built on by (most cited first, 23 of 23)

- [Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models](https://mlsanity.com/paper/2411.16602/) (2024)
- [Can Large Language Models Understand Symbolic Graphics Programs?](https://mlsanity.com/paper/2408.08313/) (2024)
- [FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement](https://mlsanity.com/paper/2503.04919/) (2025)
- [TikZero: Zero-Shot Text-Guided Graphics Program Synthesis](https://mlsanity.com/paper/2503.11509/) (2025)
- [ChatGarment: Garment Estimation, Generation and Editing via Large Language Models](https://mlsanity.com/paper/2412.17811/) (2024)
- [GRS: Generating Robotic Simulation Tasks from Real-World Images](https://mlsanity.com/paper/2410.15536/) (2024)
- [Reconstructing Animals and the Wild](https://mlsanity.com/paper/2411.18807/) (2024)
- [DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation](https://mlsanity.com/paper/2412.15200/) (2024)
- [BrickNet: Graph-Backed Generative Brick Assembly](https://mlsanity.com/paper/2604.22984/) (2026)
- [LychSim: A Controllable and Interactive Simulation Framework for Vision Research](https://mlsanity.com/paper/2605.12449/) (2026)
- [Deltaynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos](https://mlsanity.com/paper/2605.20576/) (2026)
- [4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes](https://mlsanity.com/paper/2610.03715/) (2026)
- [ZebraPose: Zebra Detection and Pose Estimation using only Synthetic Data](https://mlsanity.com/paper/2408.10831/) (2024)
- [RLS3: RL-Based Synthetic Sample Selection to Enhance Spatial Reasoning in Vision-Language Models for Indoor Autonomous Perception](https://mlsanity.com/paper/2501.18880/) (2025)
- [MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Models](https://mlsanity.com/paper/2509.22151/) (2025)
- [QE-Catalytic: A Graph-Language Multimodal Base Model for Relaxed-Energy Prediction in Catalytic Adsorption](https://mlsanity.com/paper/2512.20084/) (2025)
- [Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning](https://mlsanity.com/paper/2604.21346/) (2026)
- [3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models](https://mlsanity.com/paper/2609.06535/) (2026)
- [AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos](https://mlsanity.com/paper/2609.24487/) (2026)
- [FoldingAgent: Inferring Parametric Origami Procedures from Demonstration Videos](https://mlsanity.com/paper/2609.00377/) (2026)
- [LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction](https://mlsanity.com/paper/2610.01863/) (2026)
- [CineRecon: Benchmarking Frontier Models for Agentic Cinematic Reconstruction](https://mlsanity.com/paper/2610.16577/) (2026)
- [PhysCodeGym: Benchmarking Coding Agents for Physically Verifiable 4D Reconstruction](https://mlsanity.com/paper/2610.17288/) (2026)

## Abstract (arXiv)

Inverse graphics -- the task of inverting an image into physical variables that, when rendered, enable reproduction of the observed scene -- is a fundamental challenge in computer vision and graphics. Successfully disentangling an image into its constituent elements, such as the shape, color, and material properties of the objects of the 3D scene that produced it, requires a comprehensive understanding of the environment. This complexity limits the ability of existing carefully engineered approaches to generalize across domains. Inspired by the zero-shot ability of large language models (LLMs) to generalize to novel contexts, we investigate the possibility of leveraging the broad world knowledge encoded in such models to solve inverse-graphics problems. To this end, we propose the Inverse-Graphics Large Language Model (IG-LLM), an inverse-graphics framework centered around an LLM, that autoregressively decodes a visual embedding into a structured, compositional 3D-scene representation. We incorporate a frozen pre-trained visual encoder and a continuous numeric head to enable end-to-end training. Through our investigation, we demonstrate the potential of LLMs to facilitate inverse graphics through next-token prediction, without the application of image-space supervision. Our analysis enables new possibilities for precise spatial reasoning about images that exploit the visual knowledge of LLMs. We release our code and data at https://ig-llm.is.tue.mpg.de/ to ensure the reproducibility of our investigation and to facilitate future research.