PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion
Not on the map. Why?
Abstract (arXiv)
We present PAI-Studio, a new reference-conditioned video synthesis task that addresses a long-standing challenge in cinematic background replacement: generating dynamic backgrounds aligned with foreground motion while preserving foreground identity, matching reference scene appearance, and achieving globally consistent illumination with realistic foreground relighting. Existing open-source systems and commercial APIs cannot simultaneously ensure motion-consistent background generation, high-fidelity foreground relighting and foreground identity preservation, often resulting in static backgrounds, inconsistent boundaries, and noticeable compositing artifacts. To bridge this gap, we build upon a Diffusion Transformer video backbone and reformulate the problem as an in-context conditional generation task. Through bidirectional attention, our model jointly captures foreground dynamics and background reference information within a unified architecture. We further construct a 30K-scale dataset sourced from high-quality films and online videos to support this task. Extensive evaluations demonstrate that our method significantly outperforms existing open-source and commercial API solutions.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 58 of its 67 references that are in the map. First posted on arXiv 31 May 2026.
Builds on
58 of 67 references.
References · 58
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
- 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
- 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
- 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
- 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
- 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
38 more
- 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
- 2025SAM 3: Segment Anything with Conceptscited by 906
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
- 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
- 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
- 2025VACE: All-in-One Video Creation and Editingcited by 479
- 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
- 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
- 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
- 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
- 2018Video-to-Video Synthesiscited by 390
- 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
- 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
- 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
- 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
- 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 197
- 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 174
- 2020Real-Time High-Resolution Background Mattingcited by 173
- 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 154
- 2021Robust High-Resolution Video Matting with Temporal Guidancecited by 137
- 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
- 2024CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videoscited by 128
- 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 119
- 2024Transparent Image Layer Diffusion using Latent Transparencycited by 114
- 2022Towards An End-to-End Framework for Flow-Guided Video Inpaintingcited by 113
- 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 84
- 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNetcited by 83
- 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
- 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generationcited by 74
- 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
- 2025MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generationcited by 45
- 2025LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformercited by 43
- 2021Deep Video Matting via Spatio-Temporal Alignment and Aggregationcited by 41
- 2024Video Editing via Factorized Diffusion Distillationcited by 41
- 2024ProcessPainter: Learn Painting Process from Sequence Datacited by 38
- 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
- 2025LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuningcited by 28
- 2025IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessmentcited by 28
50 most cited of 58 All 58 (JSON)
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editingcited by 15
- 2026LiveLight: Real-time Streaming Video Relighting with Interactive Controlcited by 7
- 2026GeoEdit: Geometry-Aware Object Editing via Dual-Branch Denoisingcited by 1
- 2026MSEditor: Toward Consistent Multi-Shot Video Editingcited by 1
- 2026SubjectAnchor: Subject-Aware Memory-to-Video for Multi-Shot Storytellingcited by 1
- 2026MagicPrompt: Ultra-Lightweight Prompt Tuning for Video Generationcited by 0
- 2026Edit-VAR: Taming Visual Autoregressive Model for Precise Video Editingcited by 0
- 2026FreeCHROMA: Generative Chroma Keying from Unaligned Background Referencescited by 0
- 2026FreeCHROMA: Generative Chroma Keying from Unaligned Background References10 Oct 2026
- 2026SubjectAnchor: Subject-Aware Memory-to-Video for Multi-Shot Storytelling28 Sep 2026
- 2026Edit-VAR: Taming Visual Autoregressive Model for Precise Video Editing18 Sep 2026
- 2026MSEditor: Toward Consistent Multi-Shot Video Editing18 Aug 2026
- 2026LiveLight: Real-time Streaming Video Relighting with Interactive Control3 Aug 2026
- 2026MagicPrompt: Ultra-Lightweight Prompt Tuning for Video Generation16 Jul 2026
- 2026GeoEdit: Geometry-Aware Object Editing via Dual-Branch Denoising29 Jun 2026
- 2026LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing25 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026PAI-Actor: Cinematic Multi-Character Replacement in Dynamic Scenes
- 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generation
- 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising
- 2025ANYPORTAL: Zero-Shot Consistent Video Background Replacement
- 2024ActAnywhere: Subject-Aware Video Background Generation
- 2026LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting
- 2026Cinematic Compositing Using Character-Environment-Harmonized Video Generation Models
- 2025GenCompositor: Generative Video Compositing with Diffusion Transformer