mlsanityData to 7 Oct 2026

Text2LIVE: Text-Driven Layered Image and Video Editing

arXiv 2204.02491 ↗First posted 5 Apr 2022cs.CVcited by 312

Not on the map. Why?

Cited by 148 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a method for zero-shot, text-driven appearance manipulation in natural images and videos. Given an input image or video and a target text prompt, our goal is to edit the appearance of existing objects (e.g., object's texture) or augment the scene with visual effects (e.g., smoke, fire) in a semantically meaningful manner. We train a generator using an internal dataset of training examples, extracted from a single input (image or video and target text prompt), while leveraging an external pre-trained CLIP model to establish our losses. Rather than directly generating the edited output, our key idea is to generate an edit layer (color+opacity) that is composited over the original input. This allows us to constrain the generation process and maintain high fidelity to the original input via novel text-driven losses that are applied directly to the edit layer. Our method neither relies on a pre-trained generator nor requires user-provided edit masks. We demonstrate localized, semantic edits on high-resolution natural images and videos across a variety of objects and scenes.

Key facts. Cited by 312 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 5 Apr 2022.

Builds on

44 of 58 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
32 more
  1. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  2. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  3. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  4. 2016Generative Adversarial Text to Image Synthesiscited by 735
  5. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  6. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  7. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  8. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  9. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  10. 2021GAN Inversion: A Surveycited by 366
  11. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  12. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  13. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  14. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  15. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  16. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  17. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  18. 2017Neural Style Transfer: A Reviewcited by 199
  19. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  20. 2019Style Transfer by Relaxed Optimal Transport and Self-Similaritycited by 167
  21. 2019ManiGAN: Text-Guided Image Manipulationcited by 156
  22. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  23. 2021Paint by Wordcited by 107
  24. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  25. 2017Semantic Image Synthesis via Adversarial Learningcited by 87
  26. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  27. 2016Artistic style transfer for videoscited by 54
  28. 2020Layered Neural Rendering for Retiming People in Videocited by 51
  29. 2021Omnimatte: Associating Objects and Their Effects in Videocited by 50
  30. 2020Interactive Video Stylization Using Few-Shot Patch-Based Trainingcited by 43
  31. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 27
  32. 2017LayerBuilder: Layer Decomposition for Interactive Image and Video Color Editingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 27; 2023: 134; 2024: 90; 2025: 42; 2026: 19.

312 papers.

Sort:

By area:

  1. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  6. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  8. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  9. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  10. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  11. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  12. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
38 more
  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  2. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  3. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  4. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  5. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  6. 2022Blended Latent Diffusioncited by 430
  7. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  8. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  9. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  10. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  11. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  12. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
  13. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  14. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246
  15. 2023P+: Extended Textual Conditioning in Text-to-Image Generationcited by 242
  16. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  17. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  18. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  19. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 216
  20. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  21. 2023A Survey on Video Diffusion Modelscited by 203
  22. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  23. 2023Teaching CLIP to Count to Tencited by 194
  24. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  25. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  26. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  27. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  28. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  29. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  30. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  31. 2022SINE: SINgle Image Editing with Text-to-Image Diffusion Modelscited by 159
  32. 2023Localizing Object-level Shape Variations with Text-to-Image Diffusion Modelscited by 142
  33. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  34. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  35. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  36. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  37. 2023Delta Denoising Scorecited by 120
  38. 2023Vox-E: Text-guided Voxel Editing of 3D Objectscited by 114

50 most cited of 312 All 312 (JSON)