# Generating Procedural Materials from Text or Image Prompts

arXiv: https://arxiv.org/abs/2304.13172 · HTML: https://mlsanity.com/paper/2304.13172/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2023.

## Built on by (most cited first, 21 of 21)

- [ControlMat: A Controlled Generative Approach to Material Capture](https://mlsanity.com/paper/2309.01700/) (2023)
- [MatSynth: A Modern PBR Materials Dataset](https://mlsanity.com/paper/2401.06056/) (2024)
- [TikZero: Zero-Shot Text-Guided Graphics Program Synthesis](https://mlsanity.com/paper/2503.11509/) (2025)
- [Mesh Neural Cellular Automata](https://mlsanity.com/paper/2311.02820/) (2023)
- [VLMaterial: Procedural Material Generation with Large Vision-Language Models](https://mlsanity.com/paper/2501.18623/) (2025)
- [DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation](https://mlsanity.com/paper/2412.15200/) (2024)
- [Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians](https://mlsanity.com/paper/2412.07660/) (2024)
- [Compositional Neural Textures](https://mlsanity.com/paper/2404.12509/) (2024)
- [TexPro: Text-guided PBR Texturing with Procedural Material Modeling](https://mlsanity.com/paper/2410.15891/) (2024)
- [Generating Parametric BRDFs from Natural Language Descriptions](https://mlsanity.com/paper/2306.15679/) (2023)
- [Learning Zero-Shot Material States Segmentation, by Implanting Natural Image Patterns in Synthetic Data](https://mlsanity.com/paper/2403.03309/) (2024)
- [MaRI: Material Retrieval Integration across Domains](https://mlsanity.com/paper/2503.08111/) (2025)
- [RealMat: Realistic Materials with Diffusion and Reinforcement Learning](https://mlsanity.com/paper/2509.01134/) (2025)
- [MultiMat: Multimodal Program Synthesis for Procedural Materials using Large Multimodal Models](https://mlsanity.com/paper/2509.22151/) (2025)
- [SciTextures: Collecting and Connecting Visual Patterns, Models, and Code Across Science and Art](https://mlsanity.com/paper/2511.01817/) (2025)
- [Reflecting Process Expertise in Procedural Material Generation](https://mlsanity.com/paper/2607.13318/) (2026)
- [Scene Co-pilot: Procedural Text to Video Generation with Human in the Loop](https://mlsanity.com/paper/2411.18644/) (2024)
- [HiMat: DiT-based Ultra-High Resolution SVBRDF Generation](https://mlsanity.com/paper/2508.07011/) (2025)
- [Coding the Visual World: From Image to Simulation Using Vision Language Models](https://mlsanity.com/paper/2601.05344/) (2026)
- [FMMC: Harnessing the Power of Foundation Models for Accurate Material Classification](https://mlsanity.com/paper/2603.17390/) (2026)
- [MatLoom: Layered Text-to-Material Generation in a Compact Program Space](https://mlsanity.com/paper/2609.40322/) (2026)

## Abstract (arXiv)

Node graph systems are used ubiquitously for material design in computer graphics. They allow the use of visual programming to achieve desired effects without writing code. As high-level design tools they provide convenience and flexibility, but mastering the creation of node graphs usually requires professional training. We propose an algorithm capable of generating multiple node graphs from different types of prompts, significantly lowering the bar for users to explore a specific design space. Previous work was limited to unconditional generation of random node graphs, making the generation of an envisioned material challenging. We propose a multi-modal node graph generation neural architecture for high-quality procedural material synthesis which can be conditioned on different inputs (text or image prompts), using a CLIP-based encoder. We also create a substantially augmented material graph dataset, key to improving the generation quality. Finally, we generate high-quality graph samples using a regularized sampling process and improve the matching quality by differentiable optimization for top-ranked samples. We compare our methods to CLIP-based database search baselines (which are themselves novel) and achieve superior or similar performance without requiring massive data storage. We further show that our model can produce a set of material graphs unconditionally, conditioned on images, text prompts or partial graphs, serving as a tool for automatic visual programming completion.