mlsanityData to 7 Oct 2026

A Video is Worth 256 Bases: Spatial-Temporal Expectation-Maximization Inversion for Zero-Shot Video Editing

arXiv 2312.05856 ↗First posted 10 Dec 2023cs.CVcited by 14

On the map: Diffusion models → Video generation

Abstract (arXiv)

This paper presents a video inversion approach for zero-shot video editing, which models the input video with low-rank representation during the inversion process. The existing video editing methods usually apply the typical 2D DDIM inversion or naive spatial-temporal DDIM inversion before editing, which leverages time-varying representation for each frame to derive noisy latent. Unlike most existing approaches, we propose a Spatial-Temporal Expectation-Maximization (STEM) inversion, which formulates the dense video feature under an expectation-maximization manner and iteratively estimates a more compact basis set to represent the whole video. Each frame applies the fixed and global representation for inversion, which is more friendly for temporal consistency during reconstruction and editing. Extensive qualitative and quantitative experiments demonstrate that our STEM inversion can achieve consistent improvement on two state-of-the-art video editing methods. Project page: https://stem-inv.github.io/page/.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 48 of its 55 references that are in the map. First posted on arXiv 10 Dec 2023.

Builds on

48 of 55 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet26 Jul
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  3. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  5. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  6. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  7. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  8. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2017Neural Discrete Representation Learningcited by 4,650
26 more
  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  11. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  12. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  13. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  14. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  15. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  16. 2023Emergent Correspondence from Image Diffusioncited by 437
  17. 2022Diffusion Models already have a Semantic Latent Spacecited by 324
  18. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  19. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  20. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 290
  21. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  22. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  23. 2019Expectation-Maximization Attention Networks for Semantic Segmentationcited by 147
  24. 2023MagicEdit: High-Fidelity and Temporally Coherent Video Editingcited by 75
  25. 2023Edit-A-Video: Single Video Editing with Object-Aware Consistencycited by 65
  26. 2023Inserting Anybody in Diffusion Models via Celeb Basiscited by 63

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Splatter a Video: Video Gaussian Representation for Versatile Processingcited by 31
  2. 2024DIVE: Taming DINO for Subject-Driven Video Editingcited by 17
  3. 2024Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editingcited by 10
  4. 2024VeGaS: Video Gaussian Splattingcited by 9
  5. 2025FreeInv: Free Lunch for Improving DDIM Inversioncited by 7
  6. 2026Evolution of Video Generative Foundationscited by 6
  7. 2025Qffusion: Controllable Portrait Video Editing via Quadrant-Grid Attention Learningcited by 3
  8. 2025FAME: Fairness-aware Attention-modulated Video Editingcited by 2
  9. 2024Training-Free Video Editing via Optical Flow-Enhanced Score Distillationcited by 1
  10. 2025Follow Your Motion: A Generic Temporal Consistency Portrait Editing Framework with Trajectory Guidancecited by 1
  11. 2025Edit-Your-Interest: Efficient Video Editing via Feature Most-Similar Propagationcited by 1
  12. 2025VALA: Learning Latent Anchors for Training-Free and Temporally Consistentcited by 0
2 more
  1. 2026FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editingcited by 0
  2. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VideoDirector: Precise Video Editing via Text-to-Video Models
  2. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models
  3. 2025Zero-Shot Video Translation and Editing with Frame Spatial-Temporal Correspondence
  4. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations
  5. 2026OSVE: One Step Video Editing with One Step Diffusion Models
  6. 2025Taming Flow-based I2V Models for Creative Video Editing
  7. 2025Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversion
  8. 2026Edit-VAR: Taming Visual Autoregressive Model for Precise Video Editing