# SciTextures: Collecting and Connecting Visual Patterns, Models, and Code Across Science and Art

arXiv: https://arxiv.org/abs/2511.01817 · HTML: https://mlsanity.com/paper/2511.01817/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 50 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Builds on (most cited first, 22 of 22)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation](https://mlsanity.com/paper/2007.11301/) (2020)
- [Infinite Photorealistic Worlds using Procedural Generation](https://mlsanity.com/paper/2306.09310/) (2023)
- [Im2Vec: Synthesizing Vector Graphics without Vector Supervision](https://mlsanity.com/paper/2102.02798/) (2021)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Pre-training without Natural Images](https://mlsanity.com/paper/2101.08515/) (2021)
- [CAD-Coder: An Open-Source Vision-Language Model for Computer-Aided Design Code Generation](https://mlsanity.com/paper/2505.14646/) (2025)
- [MatSynth: A Modern PBR Materials Dataset](https://mlsanity.com/paper/2401.06056/) (2024)
- [Procedural Image Programs for Representation Learning](https://mlsanity.com/paper/2211.16412/) (2022)
- [Materialistic: Selecting Similar Materials in Images](https://mlsanity.com/paper/2305.13291/) (2023)
- [Generating Procedural Materials from Text or Image Prompts](https://mlsanity.com/paper/2304.13172/) (2023)
- [GeoCode: Interpretable Shape Programs](https://mlsanity.com/paper/2212.11715/) (2022)
- [A survey of exemplar-based texture synthesis](https://mlsanity.com/paper/1707.07184/) (2017)
- [VLMaterial: Procedural Material Generation with Large Vision-Language Models](https://mlsanity.com/paper/2501.18623/) (2025)
- [SimuGen: Multi-modal Agentic Framework for Constructing Block Diagram-Based Simulation Models](https://mlsanity.com/paper/2506.15695/) (2025)
- [Unified Modeling Language Code Generation from Diagram Images Using Multimodal Large Language Models](https://mlsanity.com/paper/2503.12293/) (2025)
- [One-shot recognition of any material anywhere using contrastive learning with physics-based rendering](https://mlsanity.com/paper/2212.00648/) (2022)
- [MetaGen: A DSL, Database, and Benchmark for VLM-Assisted Metamaterial Generation](https://mlsanity.com/paper/2508.17568/) (2025)

## Built on by (most cited first, 1 of 1)

- [Coding the Visual World: From Image to Simulation Using Vision Language Models](https://mlsanity.com/paper/2601.05344/) (2026)

## Abstract (arXiv)

The ability to connect visual patterns with the processes that form them represents one of the deepest forms of visual understanding. Textures of clouds and waves, the growth of cities and forests, or the formation of materials and landscapes are all examples of patterns emerging from underlying mechanisms. We present the SciTextures dataset, a large-scale collection of textures and visual patterns from all domains of science, tech, and art, along with the models and code that generate these images. Covering over 1,270 different models and 100,000 images of patterns and textures from physics, chemistry, biology, sociology, technology, mathematics, and art, this dataset offers a way to explore the deep connection between the visual patterns that shape our world and the mechanisms that produce them. Built through an agentic AI pipeline that autonomously collects, implements, and standardizes scientific and generative models. This AI pipeline is also used to autonomously invent and implement novel methods for generating visual patterns and textures. SciTextures enables systematic evaluation of vision language models (VLM's) ability to link visual patterns to the models and code that generate them, and to identify different patterns that emerge from the same underlying process. We also test VLMs ability to infer and recreate the mechanisms behind visual patterns by providing a natural image of a real-world phenomenon and asking the AI to identify and code a model of the process that formed it, then run this code to generate a simulated image that is compared to the reference image. These benchmarks reveal that VLM's can understand and simulate physical systems beyond visual patterns at multiple levels of abstraction. The dataset and code are available at: https://zenodo.org/records/17485502