mlsanityData to 14 Oct 2026

Watch Your Steps: Local Image and Scene Editing by Text Instructions

arXiv 2308.08947 ↗First posted 17 Aug 2023cs.CVcited by 50

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Denoising diffusion models have enabled high-quality image generation and editing. We present a method to localize the desired edit region implicit in a text instruction. We leverage InstructPix2Pix (IP2P) and identify the discrepancy between IP2P predictions with and without the instruction. This discrepancy is referred to as the relevance map. The relevance map conveys the importance of changing each pixel to achieve the edits, and is used to to guide the modifications. This guidance ensures that the irrelevant pixels remain unchanged. Relevance maps are further used to enhance the quality of text-guided editing of 3D scenes in the form of neural radiance fields. A field is trained on relevance maps of training views, denoted as the relevance field, defining the 3D region within which modifications should be made. We perform iterative updates on the training views guided by rendered relevance maps from the relevance field. Our method achieves state-of-the-art performance on both image and NeRF editing tasks. Project page: https://ashmrz.github.io/WatchYourSteps/

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 72 of its 81 references that are in the map. First posted on arXiv 17 Aug 2023.

Builds on

72 of 81 references.

Closest ancestors · 11

Same direction, newest first

  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  2. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Models17 Nov
  3. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers2 Nov
  4. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  5. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  6. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
  10. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  11. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  10. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
38 more
  1. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  2. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  3. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  4. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  5. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  6. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  7. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  9. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  10. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  11. 2022TensoRF: Tensorial Radiance Fieldscited by 1,280
  12. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  13. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 905
  14. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 735
  15. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 703
  16. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  17. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 671
  18. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 642
  19. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 600
  20. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 575
  21. 2017Convolutional Sequence to Sequence Learningcited by 552
  22. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  23. 2023Zero-shot Image-to-Image Translationcited by 505
  24. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  25. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
  26. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 468
  27. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 370
  28. 2021Advances in Neural Renderingcited by 365
  29. 2022MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architecturescited by 311
  30. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  31. 2021Editing Conditional Radiance Fieldscited by 222
  32. 2023MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenescited by 218
  33. 2022NeRF-Editing: Geometry Editing of Neural Radiance Fieldscited by 217
  34. 2021NeRF--: Neural Radiance Fields Without Known Camera Parameterscited by 155
  35. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  36. 2021BACON: Band-limited Coordinate Networks for Multiscale Scene Representationcited by 143
  37. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  38. 2023SKED: Sketch-guided Text-based 3D Editingcited by 73

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 17; 2025: 19; 2026: 5.

50 papers.

Sort:

By area:

  1. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
  2. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  3. 2023GaussianEditor: Editing 3D Gaussians Delicately with Text Instructionscited by 168
  4. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  5. 2024DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editingcited by 94
  6. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 62
  7. 2023Posterior Distillation Samplingcited by 46
  8. 2024Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directionscited by 41
  9. 2024EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splattingcited by 41
  10. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  11. 2023LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuningcited by 28
  12. 2023LIME: Localized Image Editing via Attention Regularization in Diffusion Modelscited by 27
38 more
  1. 2024AutoVFX: Physically Realistic Video Editing from Natural Language Instructionscited by 27
  2. 2024GaussianVTON: 3D Human Virtual Try-ON via Multi-Stage Gaussian Splatting Editing with Image Promptingcited by 22
  3. 2023LatentEditor: Text Driven Local Editing of 3D Scenescited by 17
  4. 2025VINCIE: Unlocking In-context Image Editing from Videocited by 17
  5. 2025Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagerycited by 17
  6. 2023Efficient-NeRF2NeRF: Streamlining Text-Driven 3D Editing with Multiview Correspondence-Enhanced Diffusion Modelscited by 16
  7. 20244Dynamic: Text-to-4D Generation with Hybrid Priorscited by 15
  8. 2024Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulationcited by 13
  9. 2024FlexEdit: Flexible and Controllable Diffusion-based Object-centric Image Editingcited by 12
  10. 2024InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenescited by 11
  11. 2024Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localizationcited by 11
  12. 2024CTRL-D: Controllable Dynamic 3D Scene Editing with Personalized 2D Diffusioncited by 11
  13. 2025Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learningcited by 9
  14. 2024DragGaussian: Enabling Drag-style Manipulation on 3D Gaussian Representationcited by 7
  15. 2025Tight Inversion: Image-Conditioned Inversion for Real Image Editingcited by 7
  16. 2025Geometry-Aware Diffusion Models for Multiview Scene Inpaintingcited by 6
  17. 2025Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolutioncited by 6
  18. 2025Balanced Image Stylization with Style Matching Scorecited by 5
  19. 2025Early Timestep Zero-Shot Candidate Selection for Instruction-Guided Image Editingcited by 5
  20. 2025Cora: Correspondence-aware image editing using few step diffusioncited by 5
  21. 2025CoreEditor: Correspondence-constrained Diffusion for Consistent 3D Editingcited by 5
  22. 2023Learning Naturally Aggregated Appearance for Efficient 3D Editingcited by 4
  23. 2025Zero-Shot Head Swapping in Real-World Scenarioscited by 4
  24. 2025POEM: Precise Object-level Editing via MLLM controlcited by 4
  25. 2026Instruction-based Image Editing with Planning, Reasoning, and Generationcited by 4
  26. 2025OmniText: A Training-Free Generalist for Controllable Text-Image Manipulationcited by 3
  27. 2024Preserving Identity with Variational Score for General-purpose 3D Editingcited by 2
  28. 2025DualNeRF: Text-Driven 3D Scene Editing via Dual-Field Representationcited by 2
  29. 2025Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infillingcited by 2
  30. 20253D-Consistent Multi-View Editing by Correspondence Guidancecited by 2
  31. 2025Towards Generalized and Training-Free Text-Guided Semantic Manipulationcited by 1
  32. 2026An Interpretable Local Editing Model for Counterfactual Medical Image Generationcited by 1
  33. 2024DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editingcited by 0
  34. 2025Mastering Regional 3DGS: Locating, Initializing, and Editing with Diverse 2D Priorscited by 0
  35. 2025InStyle: Instant Appearance Stylization of 3D Shapescited by 0
  36. 2026MIRAGE: Benchmarking and Aligning Multi-Instance Image Editingcited by 0
  37. 2026FluSplat: Sparse-View 3D Editing without Test-Time Optimizationcited by 0
  38. 2026GeM-NR: Geometry-Aware Multi-View Editing for Nonrigid Scene Changescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LocInv: Localization-aware Inversion for Text-Guided Image Editing
  2. 2025X-Edit: Detecting and Localizing Edits in Images Altered by Text-Guided Diffusion Models
  3. 2023Pix2Video: Video Editing using Image Diffusion
  4. 2025EditMGT: Unleashing Potentials of Masked Generative Transformers in Image Editing
  5. 2025DEAL-300K: Diffusion-based Editing Area Localization with a 300K-Scale Dataset and Frequency-Prompted Baseline
  6. 2024Noise Map Guidance: Inversion with Spatial Context for Real Image Editing
  7. 2025LOCATEdit: Graph Laplacian Optimized Cross Attention for Localized Text-Guided Image Editing
  8. 2025Editable Noise Map Inversion: Encoding Target-image into Noise For High-Fidelity Image Manipulation