# Object-Centric Diffusion for Efficient Video Editing

arXiv: https://arxiv.org/abs/2401.05735 · HTML: https://mlsanity.com/paper/2401.05735/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 45 of its 53 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [On Distillation of Guided Diffusion Models](https://mlsanity.com/paper/2210.03142/) (2022)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)

## Built on by (most cited first, 30 of 31)

- [Adaptive Caching for Faster Video Generation with Diffusion Transformers](https://mlsanity.com/paper/2411.02397/) (2024)
- [Motion Inversion for Video Customization](https://mlsanity.com/paper/2403.20193/) (2024)
- [Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion](https://mlsanity.com/paper/2403.14617/) (2024)
- [Importance-Based Token Merging for Efficient Image and Video Generation](https://mlsanity.com/paper/2411.16720/) (2024)
- [Mobile Video Diffusion](https://mlsanity.com/paper/2412.07583/) (2024)
- [UniVST: A Unified Framework for Training-free Localized Video Style Transfer](https://mlsanity.com/paper/2410.20084/) (2024)
- [AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration](https://mlsanity.com/paper/2412.11706/) (2024)
- [Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformer](https://mlsanity.com/paper/2509.24899/) (2025)
- [Survey of Video Diffusion Models: Foundations, Implementations, and Applications](https://mlsanity.com/paper/2504.16081/) (2025)
- [Neodragon: Mobile Video Generation using Diffusion Transformer](https://mlsanity.com/paper/2511.06055/) (2025)
- [MetaSlot: Break Through the Fixed Number of Slots in Object-Centric Learning](https://mlsanity.com/paper/2505.20772/) (2025)
- [DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation](https://mlsanity.com/paper/2511.23127/) (2025)
- [Animate Your Motion: Turning Still Images into Dynamic Videos](https://mlsanity.com/paper/2403.10179/) (2024)
- [FastVideoEdit: Leveraging Consistency Models for Efficient Text-to-Video Editing](https://mlsanity.com/paper/2403.06269/) (2024)
- [Visual Prompting for One-shot Controllable Video Editing without Inversion](https://mlsanity.com/paper/2504.14335/) (2025)
- [ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformers](https://mlsanity.com/paper/2601.04342/) (2026)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [EditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editing](https://mlsanity.com/paper/2602.15031/) (2026)
- [FreeMask: Rethinking the Importance of Attention Masks for Zero-Shot Video Editing](https://mlsanity.com/paper/2409.20500/) (2024)
- [DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation](https://mlsanity.com/paper/2606.04432/) (2026)
- [MobileWan: Closing the Quality Gap for Mobile Video Diffusion](https://mlsanity.com/paper/2607.06173/) (2026)
- [ElasticTTT: Prior-Preserving Test-Time Tuning for Video Editing](https://mlsanity.com/paper/2607.21529/) (2026)
- [Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuning](https://mlsanity.com/paper/2412.02114/) (2024)
- [FAME: Fairness-aware Attention-modulated Video Editing](https://mlsanity.com/paper/2510.22960/) (2025)
- [RFDM: Residual Flow Diffusion Model for Efficient Causal Video Editing](https://mlsanity.com/paper/2602.06871/) (2026)
- [Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing](https://mlsanity.com/paper/2502.00594/) (2025)
- [Temporal Cluster Assignment for Efficient Real-Time Video Segmentation](https://mlsanity.com/paper/2508.05851/) (2025)
- [VALA: Learning Latent Anchors for Training-Free and Temporally Consistent](https://mlsanity.com/paper/2510.22970/) (2025)
- [Importance-Aware OBS Pruning for Diffusion Models](https://mlsanity.com/paper/2607.20048/) (2026)
- [Where Compute Matters: Heterogeneous Attention for Efficient Video Diffusion](https://mlsanity.com/paper/2609.31050/) (2026)

## Abstract (arXiv)

Diffusion-based video editing have reached impressive quality and can transform either the global style, local structure, and attributes of given video inputs, following textual edit prompts. However, such solutions typically incur heavy memory and computational costs to generate temporally-coherent frames, either in the form of diffusion inversion and/or cross-frame attention. In this paper, we conduct an analysis of such inefficiencies, and suggest simple yet effective modifications that allow significant speed-ups whilst maintaining quality. Moreover, we introduce Object-Centric Diffusion, to fix generation artifacts and further reduce latency by allocating more computations towards foreground edited regions, arguably more important for perceptual quality. We achieve this by two novel proposals: i) Object-Centric Sampling, decoupling the diffusion steps spent on salient or background regions and spending most on the former, and ii) Object-Centric Token Merging, which reduces cost of cross-frame attention by fusing redundant tokens in unimportant background regions. Both techniques are readily applicable to a given video editing model without retraining, and can drastically reduce its memory and computational cost. We evaluate our proposals on inversion-based and control-signal-based editing pipelines, and show a latency reduction up to 10x for a comparable synthesis quality. Project page: qualcomm-ai-research.github.io/object-centric-diffusion.