# ZeST: Zero-Shot Material Transfer from a Single Image

arXiv: https://arxiv.org/abs/2404.06425 · HTML: https://mlsanity.com/paper/2404.06425/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 51 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023)
- [Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16322/) (2023)
- [Training-Free Layout Control with Cross-Attention Guidance](https://mlsanity.com/paper/2304.03373/) (2023)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [P+: Extended Textual Conditioning in Text-to-Image Generation](https://mlsanity.com/paper/2303.09522/) (2023)
- [Text2Tex: Text-driven Texture Synthesis via Diffusion Models](https://mlsanity.com/paper/2303.11396/) (2023)
- [Subject-driven Text-to-Image Generation via Apprenticeship Learning](https://mlsanity.com/paper/2304.00186/) (2023)
- [ReCo: Region-Controlled Text-to-Image Generation](https://mlsanity.com/paper/2211.15518/) (2022)

## Built on by (most cited first, 24 of 24)

- [IntrinsicEdit: Precise generative image manipulation in intrinsic space](https://mlsanity.com/paper/2505.08889/) (2025)
- [Conditional Balance: Improving Multi-Conditioning Trade-Offs in Image Generation](https://mlsanity.com/paper/2412.19853/) (2024)
- [MARBLE: Material Recomposition and Blending in CLIP-Space](https://mlsanity.com/paper/2506.05313/) (2025)
- [Eye-for-an-eye: Appearance Transfer with Semantic Correspondence in Diffusion Models](https://mlsanity.com/paper/2406.07008/) (2024)
- [MatSwap: Light-aware material transfers in images](https://mlsanity.com/paper/2502.07784/) (2025)
- [LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model](https://mlsanity.com/paper/2412.11519/) (2024)
- [EditCLIP: Representation Learning for Image Editing](https://mlsanity.com/paper/2503.20318/) (2025)
- [Controllable-Continuous Color Editing in Diffusion Model via Color Mapping](https://mlsanity.com/paper/2509.13756/) (2025)
- [MaRI: Material Retrieval Integration across Domains](https://mlsanity.com/paper/2503.08111/) (2025)
- [A Controllable Appearance Representation for Flexible Transfer and Editing](https://mlsanity.com/paper/2504.15028/) (2025)
- [PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation](https://mlsanity.com/paper/2504.17067/) (2025)
- [RETRO: REthinking Tactile Representation Learning with Material PriOrs](https://mlsanity.com/paper/2505.14319/) (2025)
- [TexEditor: Structure-Preserving Text-Driven Texture Editing](https://mlsanity.com/paper/2603.18488/) (2026)
- [Inverse Image-Based Rendering for Light Field Generation from Single Images](https://mlsanity.com/paper/2510.20132/) (2025)
- [CoatFusion: Controllable Material Coating in Images](https://mlsanity.com/paper/2512.02143/) (2025)
- [FROMAT: Multiview Material Appearance Transfer via Few-Shot Self-Attention Adaptation](https://mlsanity.com/paper/2512.09617/) (2025)
- [Alterbute: Editing Intrinsic Attributes of Objects in Images](https://mlsanity.com/paper/2601.10714/) (2026)
- [FMMC: Harnessing the Power of Foundation Models for Accurate Material Classification](https://mlsanity.com/paper/2603.17390/) (2026)
- [A training-free framework for high-fidelity appearance transfer via diffusion transformers](https://mlsanity.com/paper/2603.26767/) (2026)
- [DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer](https://mlsanity.com/paper/2605.15681/) (2026)
- [Controllable Texture Tiling with Transformed RoPE-Enhanced Diffusion Models](https://mlsanity.com/paper/2606.22945/) (2026)
- [AISPO: Enhancing Depth Reliability for Robotic Manipulation of Non-Lambertian Objects via Affine-Invariant Shape Prior](https://mlsanity.com/paper/2606.25503/) (2026)
- [MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes](https://mlsanity.com/paper/2608.24107/) (2026)
- [LoGAN: Multilingual Font Localization with Generative Agents](https://mlsanity.com/paper/2609.07029/) (2026)

## Abstract (arXiv)

We propose ZeST, a method for zero-shot material transfer to an object in the input image given a material exemplar image. ZeST leverages existing diffusion adapters to extract implicit material representation from the exemplar image. This representation is used to transfer the material using pre-trained inpainting diffusion model on the object in the input image using depth estimates as geometry cue and grayscale object shading as illumination cues. The method works on real images without any training resulting a zero-shot approach. Both qualitative and quantitative results on real and synthetic datasets demonstrate that ZeST outputs photorealistic images with transferred materials. We also show the application of ZeST to perform multiple edits and robust material assignment under different illuminations. Project Page: https://ttchengab.github.io/zest