mlsanityData to 7 Oct 2026

Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuning

arXiv 2412.02114 ↗First posted 3 Dec 2024cs.CV cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Recent advances in video generation have outpaced progress in video editing, which remains constrained by several limiting factors, namely: (a) the task's dependency on supervision severely limits generality, (b) an unnecessary artificial separation between the generation and editing task, and (c) the high computational costs of training a video model. In this work, we propose UES (Unlocking Universal Editing via Self-Supervision), a lightweight self-supervised fine-tuning strategy that transforms generation models into unified generation-editing systems through self-supervised semantic alignment. Our approach establishes a dual-conditioning mechanism where original video-text pairs jointly provide visual and textual semantics, enabling structured learning of intrinsic spatiotemporal correspondences. Key advantages include: (i) Universality through supervision-free adaptation to diverse editing tasks, (ii) Unification of generation and editing applicable to most text(+image)-to-video model, and (iii) Efficiency via lightweight fine-tune that reduces tunable parameters by 92.67%. To enable systematic evaluation, we introduce OmniBench-99, a comprehensive benchmark spanning 99 videos across humans/animals, environments, and objects, comprising 4 editing types and 8 scenarios. Extensive experiments show UES enables models without inherent editing capability to perform powerful and universal editing while preserving or even enhancing their original generation performance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 106 of its 122 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

106 of 122 references.

References · 106

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
38 more
  1. 2022Video Diffusion Modelscited by 2,280
  2. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  8. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  9. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  11. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  13. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  14. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  15. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  16. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  17. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  18. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  19. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  20. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  21. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  22. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  23. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  24. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  25. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  26. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  27. 2024OmniGen: Unified Image Generationcited by 388
  28. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  29. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
  30. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  31. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  32. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  33. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  34. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  35. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  36. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  37. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  38. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.