# LayerFlow: A Unified Model for Layer-aware Video Generation

arXiv: https://arxiv.org/abs/2506.04228 · HTML: https://mlsanity.com/paper/2506.04228/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion](https://mlsanity.com/paper/2402.03162/) (2024)
- [TrailBlazer: Trajectory Control for Diffusion-Based Video Generation](https://mlsanity.com/paper/2401.00896/) (2023)
- [AVID: Any-Length Video Inpainting with Diffusion Model](https://mlsanity.com/paper/2312.03816/) (2023)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [Still-Moving: Customized Video Generation without Customized Video Data](https://mlsanity.com/paper/2407.08674/) (2024)
- [UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing](https://mlsanity.com/paper/2402.13185/) (2024)
- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [TransAnimate: Taming Layer Diffusion to Generate RGBA Video](https://mlsanity.com/paper/2503.17934/) (2025)
- [Alfie: Democratising RGBA Image Generation With No ](https://mlsanity.com/paper/2408.14826/) (2024)
- [TransPixeler: Advancing Text-to-Video Generation with Transparency](https://mlsanity.com/paper/2501.03006/) (2025)
- [FactorMatte: Redefining Video Matting for Re-Composition Tasks](https://mlsanity.com/paper/2211.02145/) (2022)

## Built on by (most cited first, 10 of 10)

- [SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation](https://mlsanity.com/paper/2506.23690/) (2025)
- [Video Generation with Stable Transparency via Shiftable RGB-A Distribution Learner](https://mlsanity.com/paper/2509.24979/) (2025)
- [CoDance: An Unbind-Rebind Paradigm for Robust Multi-Subject Animation](https://mlsanity.com/paper/2601.11096/) (2026)
- [LayerT2V: A Unified Multi-Layer Video Generation Framework](https://mlsanity.com/paper/2508.04228/) (2025)
- [FastDDHPose: Towards Unified, Efficient, and Disentangled 3D Human Pose Estimation](https://mlsanity.com/paper/2512.14162/) (2025)
- [TransText: Alpha-as-RGB Representation for Transparent Text Animation](https://mlsanity.com/paper/2603.17944/) (2026)
- [Layer-Aware Video Composition via Split-then-Merge](https://mlsanity.com/paper/2511.20809/) (2025)
- [LimeCross: Context-Conditioned Layered Image Editing with Structural Consistency](https://mlsanity.com/paper/2605.10319/) (2026)
- [Vera: A Layered Diffusion Model for Content-Preserving Video Editing](https://mlsanity.com/paper/2606.23610/) (2026)
- [Alpha as an Efficiency Signal: Visibility-Routed RGBA Image-to-Video Generation](https://mlsanity.com/paper/2608.09355/) (2026)

## Abstract (arXiv)

We present LayerFlow, a unified solution for layer-aware video generation. Given per-layer prompts, LayerFlow generates videos for the transparent foreground, clean background, and blended scene. It also supports versatile variants like decomposing a blended video or generating the background for the given foreground and vice versa. Starting from a text-to-video diffusion transformer, we organize the videos for different layers as sub-clips, and leverage layer embeddings to distinguish each clip and the corresponding layer-wise prompts. In this way, we seamlessly support the aforementioned variants in one unified framework. For the lack of high-quality layer-wise training videos, we design a multi-stage training strategy to accommodate static images with high-quality layer annotations. Specifically, we first train the model with low-quality video data. Then, we tune a motion LoRA to make the model compatible with static frames. Afterward, we train the content LoRA on the mixture of image data with high-quality layered images along with copy-pasted video data. During inference, we remove the motion LoRA thus generating smooth videos with desired layers.