# Transparent Image Layer Diffusion using Latent Transparency

arXiv: https://arxiv.org/abs/2402.17113 · HTML: https://mlsanity.com/paper/2402.17113/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 48 of its 73 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 48)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Imagic: Text-Based Real Image Editing with Diffusion Models](https://mlsanity.com/paper/2210.09276/) (2022)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Blended Diffusion for Text-driven Editing of Natural Images](https://mlsanity.com/paper/2111.14818/) (2021)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [InstantID: Zero-shot Identity-Preserving Generation in Seconds](https://mlsanity.com/paper/2401.07519/) (2024)
- [Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2305.16322/) (2023)
- [PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding](https://mlsanity.com/paper/2312.04461/) (2023)

## Built on by (most cited first, 30 of 114)

- [OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion](https://mlsanity.com/paper/2507.06165/) (2025)
- [LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation](https://mlsanity.com/paper/2408.13252/) (2024)
- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [BooW-VTON: Boosting In-the-Wild Virtual Try-On via Mask-Free Pseudo Data Training](https://mlsanity.com/paper/2408.06047/) (2024)
- [OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Data](https://mlsanity.com/paper/2501.07397/) (2025)
- [ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation](https://mlsanity.com/paper/2502.18364/) (2025)
- [GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priors](https://mlsanity.com/paper/2504.01016/) (2025)
- [CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic Design](https://mlsanity.com/paper/2505.19114/) (2025)
- [Generative Image Layer Decomposition with Visual Effects](https://mlsanity.com/paper/2411.17864/) (2024)
- [Qwen-Image-Layered: Towards Inherent Editability via Layer Decomposition](https://mlsanity.com/paper/2512.15603/) (2025)
- [Dual-Interrelated Diffusion Model for Few-Shot Anomaly Image Generation](https://mlsanity.com/paper/2408.13509/) (2024)
- [HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editing](https://mlsanity.com/paper/2412.04280/) (2024)
- [Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation](https://mlsanity.com/paper/2503.16430/) (2025)
- [RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformers](https://mlsanity.com/paper/2506.02528/) (2025)
- [CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models](https://mlsanity.com/paper/2404.15677/) (2024)
- [TransAnimate: Taming Layer Diffusion to Generate RGBA Video](https://mlsanity.com/paper/2503.17934/) (2025)
- [Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches](https://mlsanity.com/paper/2408.04567/) (2024)
- [Alfie: Democratising RGBA Image Generation With No ](https://mlsanity.com/paper/2408.14826/) (2024)
- [BizGen: Advancing Article-level Visual Text Rendering for Infographics Generation](https://mlsanity.com/paper/2503.20672/) (2025)
- [DiffDecompose: Layer-Wise Decomposition of Alpha-Composited Images via Diffusion Transformers](https://mlsanity.com/paper/2505.21541/) (2025)
- [Open-World Amodal Appearance Completion](https://mlsanity.com/paper/2411.13019/) (2024)
- [LayerFusion: Harmonized Multi-Layer Text-to-Image Generation with Generative Priors](https://mlsanity.com/paper/2412.04460/) (2024)
- [CreatiPoster: Towards Editable and Controllable Multi-Layer Graphic Design Generation](https://mlsanity.com/paper/2506.10890/) (2025)
- [Layered Image Vectorization via Semantic Simplification](https://mlsanity.com/paper/2406.05404/) (2024)
- [Diffusion-Based Visual Art Creation: A Survey and New Perspectives](https://mlsanity.com/paper/2408.12128/) (2024)
- [DreamLayer: Simultaneous Multi-Layer Generation via Diffusion Mode](https://mlsanity.com/paper/2503.12838/) (2025)
- [PrismLayers: Open Data for High-Quality Multi-Layer Transparent Image Generative Models](https://mlsanity.com/paper/2505.22523/) (2025)
- [Generating Compositional Scenes via Text-to-image RGBA Instance Generation](https://mlsanity.com/paper/2411.10913/) (2024)
- [OmniPSD: Layered PSD Generation with Diffusion Transformer](https://mlsanity.com/paper/2512.09247/) (2025)
- [TransPixeler: Advancing Text-to-Video Generation with Transparency](https://mlsanity.com/paper/2501.03006/) (2025)

## Abstract (arXiv)

We present LayerDiffuse, an approach enabling large-scale pretrained latent diffusion models to generate transparent images. The method allows generation of single transparent images or of multiple transparent layers. The method learns a "latent transparency" that encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. It preserves the production-ready quality of the large diffusion model by regulating the added transparency as a latent offset with minimal changes to the original latent distribution of the pretrained model. In this way, any latent diffusion model can be converted into a transparent image generator by finetuning it with the adjusted latent space. We train the model with 1M transparent image layer pairs collected using a human-in-the-loop collection scheme. We show that latent transparency can be applied to different open source image generators, or be adapted to various conditional control systems to achieve applications like foreground/background-conditioned layer generation, joint layer generation, structural control of layer contents, etc. A user study finds that in most cases (97%) users prefer our natively generated transparent content over previous ad-hoc solutions such as generating and then matting. Users also report the quality of our generated transparent images is comparable to real commercial transparent assets like Adobe Stock.