A Controllable Appearance Representation for Flexible Transfer and Editing
Not on the map. Why?
Abstract (arXiv)
We present a method that computes an interpretable representation of material appearance within a highly compact, disentangled latent space. This representation is learned in a self-supervised fashion using an adapted FactorVAE. We train our model with a carefully designed unlabeled dataset, avoiding possible biases induced by human-generated labels. Our model demonstrates strong disentanglement and interpretability by effectively encoding material appearance and illumination, despite the absence of explicit supervision. Then, we use our representation as guidance for training a lightweight IP-Adapter to condition a diffusion pipeline that transfers the appearance of one or more images onto a target geometry, and allows the user to further edit the resulting appearance. Our approach offers fine-grained control over the generated results: thanks to the well-structured compact latent space, users can intuitively manipulate attributes such as hue or glossiness in image space to achieve the desired final appearance.
Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 39 of its 66 references that are in the map. First posted on arXiv 21 Apr 2025.
Builds on
39 of 66 references.
References · 39
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
- 2020Denoising Diffusion Probabilistic Modelscited by 20,952
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,082
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,000
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,484
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,323
- 2015Fast R-CNNcited by 2,933
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,604
27 more
- 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,453
- 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,342
- 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 934
- 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 860
- 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 325
- 2016Ladder Variational Autoencoderscited by 318
- 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
- 2015Deep Convolutional Inverse Graphics Networkcited by 121
- 2018Single-Image SVBRDF Capture with a Rendering-Aware Deep Networkcited by 116
- 2022Disentangled Representation Learningcited by 112
- 2023ControlMat: A Controlled Generative Approach to Material Capturecited by 50
- 2024DreamMat: High-quality PBR Material Generation with Geometry- and Light-aware Diffusion Modelscited by 50
- 2023Alchemist: Parametric Control of Material Properties with Diffusion Modelscited by 38
- 2022Disentanglement with Biological Constraints: A Theory of Functional Cell Typescited by 30
- 2023DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Modelscited by 30
- 2024ColorPeel: Color Prompt Learning with Diffusion Models via Color and Shape Disentanglementcited by 28
- 2024ZeST: Zero-Shot Material Transfer from a Single Imagecited by 24
- 2024TexSliders: Diffusion-Based Texture Editing in CLIP Spacecited by 23
- 2024Collaborative Control for Geometry-Conditioned PBR Image Generationcited by 22
- 2019A Similarity Measure for Material Appearancecited by 19
- 2019Re-balancing Variational Autoencoder Loss for Molecule Sequence Generationcited by 12
- 2018An intuitive control space for material appearancecited by 10
- 2023In-the-wild Material Appearance Editing using Perceptual Attributescited by 9
- 2023Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Networkcited by 7
- 2021The joint role of geometry and illumination on material recognitioncited by 5
- 2022Interpretable Disentangled Parametrization of Measured BRDF with β-VAEcited by 3
- 2024Predicting Perceived Gloss: Do Weak Labels Suffice?cited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.