# PhotoMat: A Material Generator Learned from Single Flash Photos

arXiv: https://arxiv.org/abs/2305.12296 · HTML: https://mlsanity.com/paper/2305.12296/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Single-Image SVBRDF Capture with a Rendering-Aware Deep Network](https://mlsanity.com/paper/1810.09718/) (2018)
- [Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image](https://mlsanity.com/paper/1804.05790/) (2018)
- [Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks](https://mlsanity.com/paper/1809.00886/) (2018)
- [Flexible SVBRDF Capture with a Multi-Image Deep Network](https://mlsanity.com/paper/1906.11557/) (2019)
- [Generative Modelling of BRDF Textures from Flash Images](https://mlsanity.com/paper/2102.11861/) (2021)
- [MaterialGAN: Reflectance Capture using a Generative SVBRDF Model](https://mlsanity.com/paper/2010.00114/) (2020)
- [Guided Fine-Tuning for Large-Scale Material Transfer](https://mlsanity.com/paper/2007.03059/) (2020)
- [TileGen: Tileable, Controllable Material Generation and Capture](https://mlsanity.com/paper/2206.05649/) (2022)
- [NeuMIP: Multi-Resolution Neural Materials](https://mlsanity.com/paper/2104.02789/) (2021)
- [Controlling Material Appearance by Examples](https://mlsanity.com/paper/2206.14970/) (2022)

## Built on by (most cited first, 18 of 18)

- [ControlMat: A Controlled Generative Approach to Material Capture](https://mlsanity.com/paper/2309.01700/) (2023)
- [MatSynth: A Modern PBR Materials Dataset](https://mlsanity.com/paper/2401.06056/) (2024)
- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [VLMaterial: Procedural Material Generation with Large Vision-Language Models](https://mlsanity.com/paper/2501.18623/) (2025)
- [Generative Escher Meshes](https://mlsanity.com/paper/2309.14564/) (2023)
- [Chord: Chain of Rendering Decomposition for PBR Material Estimation from Generated Texture Images](https://mlsanity.com/paper/2509.09952/) (2025)
- [OLATverse: A Large-scale Real-world Object Dataset with Precise Lighting Control](https://mlsanity.com/paper/2511.02483/) (2025)
- [StableMaterials: Enhancing Diversity in Material Generation via Semi-Supervised Learning](https://mlsanity.com/paper/2406.09293/) (2024)
- [Generating Parametric BRDFs from Natural Language Descriptions](https://mlsanity.com/paper/2306.15679/) (2023)
- [MatPedia: A Universal Generative Foundation for High-Fidelity Material Synthesis](https://mlsanity.com/paper/2511.16957/) (2025)
- [One Noise to Rule Them All: Learning a Unified Model of Spatially-Varying Noise Patterns](https://mlsanity.com/paper/2404.16292/) (2024)
- [MaRI: Material Retrieval Integration across Domains](https://mlsanity.com/paper/2503.08111/) (2025)
- [VideoNeuMat: Neural Material Extraction from Generative Video Models](https://mlsanity.com/paper/2602.07272/) (2026)
- [RealMat: Realistic Materials with Diffusion and Reinforcement Learning](https://mlsanity.com/paper/2509.01134/) (2025)
- [Neural Appearance Modeling From Single Images](https://mlsanity.com/paper/2406.18593/) (2024)
- [HiMat: DiT-based Ultra-High Resolution SVBRDF Generation](https://mlsanity.com/paper/2508.07011/) (2025)
- [SpecGen: Neural Spectral BRDF Generation via Spectral-Spatial Tri-plane Aggregation](https://mlsanity.com/paper/2508.17316/) (2025)
- [FMMC: Harnessing the Power of Foundation Models for Accurate Material Classification](https://mlsanity.com/paper/2603.17390/) (2026)

## Abstract (arXiv)

Authoring high-quality digital materials is key to realism in 3D rendering. Previous generative models for materials have been trained exclusively on synthetic data; such data is limited in availability and has a visual gap to real materials. We circumvent this limitation by proposing PhotoMat: the first material generator trained exclusively on real photos of material samples captured using a cell phone camera with flash. Supervision on individual material maps is not available in this setting. Instead, we train a generator for a neural material representation that is rendered with a learned relighting module to create arbitrarily lit RGB images; these are compared against real photos using a discriminator. We then train a material maps estimator to decode material reflectance properties from the neural material representation. We train PhotoMat with a new dataset of 12,000 material photos captured with handheld phone cameras under flash lighting. We demonstrate that our generated materials have better visual quality than previous material generators trained on synthetic data. Moreover, we can fit analytical material models to closely match these generated neural materials, thus allowing for further editing and use in 3D rendering.