# BlenderAlchemy: Editing 3D Graphics with Vision-Language Models

arXiv: https://arxiv.org/abs/2404.17672 · HTML: https://mlsanity.com/paper/2404.17672/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 49 of its 69 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 49)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023)
- [Text2Tex: Text-driven Texture Synthesis via Diffusion Models](https://mlsanity.com/paper/2303.11396/) (2023)
- [Woodpecker: Hallucination Correction for Multimodal Large Language Models](https://mlsanity.com/paper/2310.16045/) (2023)
- [Foundation Models in Robotics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2312.07843/) (2023)
- [Is Self-Repair a Silver Bullet for Code Generation?](https://mlsanity.com/paper/2306.09896/) (2023)
- [GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation](https://mlsanity.com/paper/2401.04092/) (2024)
- [Infinite Photorealistic Worlds using Procedural Generation](https://mlsanity.com/paper/2306.09310/) (2023)
- [Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models](https://mlsanity.com/paper/2312.13913/) (2023)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [3D-GPT: Procedural 3D Modeling with Large Language Models](https://mlsanity.com/paper/2310.12945/) (2023)

## Built on by (most cited first, 22 of 22)

- [BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing](https://mlsanity.com/paper/2504.01786/) (2025)
- [GRS: Generating Robotic Simulation Tasks from Real-World Images](https://mlsanity.com/paper/2410.15536/) (2024)
- [VLMgineer: Vision Language Models as Robotic Toolsmiths](https://mlsanity.com/paper/2507.12644/) (2025)
- [Towards Scalable and Consistent 3D Editing](https://mlsanity.com/paper/2510.02994/) (2025)
- [VLMaterial: Procedural Material Generation with Large Vision-Language Models](https://mlsanity.com/paper/2501.18623/) (2025)
- [SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation](https://mlsanity.com/paper/2607.01766/) (2026)
- [BlenderFusion: 3D-Grounded Visual Editing and Generative Compositing](https://mlsanity.com/paper/2506.17450/) (2025)
- [Affine-Equivariant Kernel Space Encoding for NeRF Editing](https://mlsanity.com/paper/2508.02831/) (2025)
- [MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs](https://mlsanity.com/paper/2607.08970/) (2026)
- [MetaGen: A DSL, Database, and Benchmark for VLM-Assisted Metamaterial Generation](https://mlsanity.com/paper/2508.17568/) (2025)
- [SceneConductor: 3D Scene Generation from a Single Image with Multi-Agent Orchestration](https://mlsanity.com/paper/2606.08402/) (2026)
- [ScribbleSense: Generative Scribble-Based Texture Editing with Intent Prediction](https://mlsanity.com/paper/2601.22455/) (2026)
- [3D-CoS: A New 3D Reconstruction Paradigm Based on VLM Code Synthesis](https://mlsanity.com/paper/2606.10478/) (2026)
- [Reflecting Process Expertise in Procedural Material Generation](https://mlsanity.com/paper/2607.13318/) (2026)
- [MaLiang-Harness: A Programmable Path to Image and Video Generation](https://mlsanity.com/paper/2609.34309/) (2026)
- [Handle-based Mesh Deformation Guided By Vision Language Model](https://mlsanity.com/paper/2506.04562/) (2025)
- [ProcFunc: Function-Oriented Abstractions for Procedural 3D Generation in Python](https://mlsanity.com/paper/2604.26943/) (2026)
- [LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction](https://mlsanity.com/paper/2610.01863/) (2026)
- [World Editing: Intervening on Executable Worlds at Increasing Depth](https://mlsanity.com/paper/2610.02331/) (2026)
- [VETR: Execution-Grounded Self-Improving Agent for Symbolic Graphics Reasoning](https://mlsanity.com/paper/2610.13098/) (2026)
- [ViewStruct: View-Grounded Articulation and Connectivity Inference](https://mlsanity.com/paper/2610.16793/) (2026)
- [LAMP: 3D Source-Aware Lighting Estimation at Multiple Points](https://mlsanity.com/paper/2610.19266/) (2026)

## Abstract (arXiv)

Graphics design is important for various applications, including movie production and game design. To create a high-quality scene, designers usually need to spend hours in software like Blender, in which they might need to interleave and repeat operations, such as connecting material nodes, hundreds of times. Moreover, slightly different design goals may require completely different sequences, making automation difficult. In this paper, we propose a system that leverages Vision-Language Models (VLMs), like GPT-4V, to intelligently search the design action space to arrive at an answer that can satisfy a user's intent. Specifically, we design a vision-based edit generator and state evaluator to work together to find the correct sequence of actions to achieve the goal. Inspired by the role of visual imagination in the human design process, we supplement the visual reasoning capabilities of VLMs with "imagined" reference images from image-generation models, providing visual grounding of abstract language descriptions. In this paper, we provide empirical evidence suggesting our system can produce simple but tedious Blender editing sequences for tasks such as editing procedural materials and geometry from text and/or reference images, as well as adjusting lighting configurations for product renderings in complex scenes.