Coding the Visual World: From Image to Simulation Using Vision Language Models
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
The ability to construct mental models of the world is a central aspect of understanding. Similarly, visual understanding can be viewed as the ability to construct a representative model of the system depicted in an image. This work explores the capacity of Vision Language Models (VLMs) to recognize and simulate the systems and mechanisms depicted in images using the Im2Sim methodology. The VLM is given a natural image of a real-world system (e.g., cities, clouds, vegetation) and is tasked with describing the system and writing code that simulates and generates it. This generative code is then executed to produce a synthetic image, which is compared against the original. This approach is tested on various complex emergent systems, ranging from physical systems (waves, lights, clouds) to vegetation, cities, materials, and geological formations. Through analysis of the models and images generated by the VLMs, we examine their understanding of the systems in images. The results show that leading VLMs (GPT, Gemini) have the ability to understand and model complex, multi-component systems across multiple layers of abstraction and a wide range of domains. At the same time, the VLMs exhibit limited ability to replicate fine details and low-level arrangements of patterns in the image. These findings reveal an interesting asymmetry: VLMs combine high-level, deep visual understanding of images with limited perception of fine details.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 32 references that are in the map. First posted on arXiv 8 Jan 2026.
Builds on
16 of 32 references.
References · 16
- 2023Visual Instruction Tuningcited by 7,869
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
- 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
- 2021DeepCAD: A Deep Generative Network for Computer-Aided Design Modelscited by 271
- 2020DeepSVG: A Hierarchical Generative Network for Vector Graphics Animationcited by 159
- 2021Im2Vec: Synthesizing Vector Graphics without Vector Supervisioncited by 116
- 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 114
- 2025CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generationcited by 43
- 2023Generating Procedural Materials from Text or Image Promptscited by 21
- 2022GeoCode: Interpretable Shape Programscited by 16
- 2025VLMaterial: Procedural Material Generation with Large Vision-Language Modelscited by 9
4 more
- 2025SimuGen: Multi-modal Agentic Framework for Constructing Block Diagram-Based Simulation Modelscited by 8
- 2025Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Modelscited by 7
- 2025Shape and Texture Recognition in Large Vision-Language Modelscited by 3
- 2025SciTextures: Collecting and Connecting Visual Patterns, Models, and Code Across Science and Artcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024An Introduction to Vision-Language Modeling
- 2025VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation
- 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation
- 2025Can World Models Benefit VLMs for World Dynamics?
- 2025Adapting Vision-Language Models for Evaluating World Models
- 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectives
- 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning
- 2025Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision