mlsanityData to 14 Oct 2026

PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion

arXiv 2606.01399 ↗First posted 31 May 2026cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

We present PAI-Studio, a new reference-conditioned video synthesis task that addresses a long-standing challenge in cinematic background replacement: generating dynamic backgrounds aligned with foreground motion while preserving foreground identity, matching reference scene appearance, and achieving globally consistent illumination with realistic foreground relighting. Existing open-source systems and commercial APIs cannot simultaneously ensure motion-consistent background generation, high-fidelity foreground relighting and foreground identity preservation, often resulting in static backgrounds, inconsistent boundaries, and noticeable compositing artifacts. To bridge this gap, we build upon a Diffusion Transformer video backbone and reformulate the problem as an in-context conditional generation task. Through bidirectional attention, our model jointly captures foreground dynamics and background reference information within a unified architecture. We further construct a 30K-scale dataset sourced from high-quality films and online videos to support this task. Extensive evaluations demonstrate that our method significantly outperforms existing open-source and commercial API solutions.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 58 of its 67 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

58 of 67 references.

References · 58

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2025Qwen3-VL Technical Reportcited by 2,793
  5. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  6. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  9. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
38 more
  1. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  2. 2025SAM 3: Segment Anything with Conceptscited by 906
  3. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  5. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  6. 2025VACE: All-in-One Video Creation and Editingcited by 479
  7. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  8. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  9. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  10. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  11. 2018Video-to-Video Synthesiscited by 390
  12. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  13. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  14. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  15. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  16. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 197
  17. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 174
  18. 2020Real-Time High-Resolution Background Mattingcited by 173
  19. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 154
  20. 2021Robust High-Resolution Video Matting with Temporal Guidancecited by 137
  21. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  22. 2024CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videoscited by 128
  23. 2025EasyControl: Adding Efficient and Flexible Control for Diffusion Transformercited by 119
  24. 2024Transparent Image Layer Diffusion using Latent Transparencycited by 114
  25. 2022Towards An End-to-End Framework for Flow-Guided Video Inpaintingcited by 113
  26. 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 84
  27. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNetcited by 83
  28. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  29. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generationcited by 74
  30. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
  31. 2025MakeAnything: Harnessing Diffusion Transformers for Multi-Domain Procedural Sequence Generationcited by 45
  32. 2025LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformercited by 43
  33. 2021Deep Video Matting via Spatio-Temporal Alignment and Aggregationcited by 41
  34. 2024Video Editing via Factorized Diffusion Distillationcited by 41
  35. 2024ProcessPainter: Learn Painting Process from Sequence Datacited by 38
  36. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  37. 2025LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuningcited by 28
  38. 2025IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessmentcited by 28

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026PAI-Actor: Cinematic Multi-Character Replacement in Dynamic Scenes
  2. 2025Frame In-N-Out: Unbounded Controllable Image-to-Video Generation
  3. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising
  4. 2025ANYPORTAL: Zero-Shot Consistent Video Background Replacement
  5. 2024ActAnywhere: Subject-Aware Video Background Generation
  6. 2026LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting
  7. 2026Cinematic Compositing Using Character-Environment-Harmonized Video Generation Models
  8. 2025GenCompositor: Generative Video Compositing with Diffusion Transformer