mlsanityData to 14 Oct 2026

Mapping the Mind of an Instruction-based Image Editing using SMILE

arXiv 2412.16277 ↗First posted 20 Dec 2024cs.AI cs.CV cs.HCcited by 3

Not on the map. Why?

Abstract (arXiv)

Despite recent advancements in Instruct-based Image Editing models for generating high-quality images, they are known as black boxes and a significant barrier to transparency and user trust. To solve this issue, we introduce SMILE (Statistical Model-agnostic Interpretability with Local Explanations), a novel model-agnostic for localized interpretability that provides a visual heatmap to clarify the textual elements' influence on image-generating models. We applied our method to various Instruction-based Image Editing models like Pix2Pix, Image2Image-turbo and Diffusers-Inpaint and showed how our model can improve interpretability and reliability. Also, we use stability, accuracy, fidelity, and consistency metrics to evaluate our method. These findings indicate the exciting potential of model-agnostic interpretability for reliability and trustworthiness in critical applications such as healthcare and autonomous driving while encouraging additional investigation into the significance of interpretability in enhancing dependable image editing models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 66 of its 109 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

66 of 109 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  8. 2022Generative Adversarial Networkscited by 5,627
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  12. 2017Improved Training of Wasserstein GANscited by 2,235
38 more
  1. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  3. 2022Diffusion Models in Vision: A Surveycited by 1,015
  4. 2016End to End Learning for Self-Driving Carscited by 1,009
  5. 2018A Survey Of Methods For Explaining Black Box Modelscited by 973
  6. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  7. 2017Wasserstein GANcited by 639
  8. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  9. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 612
  10. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  11. 2019Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methodscited by 443
  12. 2020Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlationcited by 425
  13. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  14. 2016A Network-based End-to-End Trainable Task-oriented Dialogue Systemcited by 252
  15. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 236
  16. 2013Fast Computation of Wasserstein Barycenterscited by 216
  17. 2020GraphLIME: Local Interpretable Model Explanations for Graph Neural Networkscited by 209
  18. 2017Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Modelscited by 206
  19. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  20. 2017What do we need to build explainable AI systems for the medical domain?cited by 129
  21. 2024One-Step Image Translation with Text-to-Image Modelscited by 116
  22. 2020Explaining the Explainer: A First Theoretical Analysis of LIMEcited by 89
  23. 2023The Metacognitive Demands and Opportunities of Generative AIcited by 87
  24. 2018Model Agnostic Supervised Local Explanationscited by 83
  25. 2023Diffusion Models for Open-Vocabulary Segmentationcited by 64
  26. 2023Focus on Your Instruction: Fine-grained and Multi-instruction Image Editing by Attention Modulationcited by 62
  27. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 62
  28. 2020Artificial Intelligence in the Creative Industries: A Reviewcited by 61
  29. 2021S-LIME: Stabilized-LIME for Model Explanationcited by 52
  30. 2024A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Modelscited by 52
  31. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  32. 2020BayLIME: Bayesian Local Interpretable Model-Agnostic Explanationscited by 51
  33. 2022Efficient Diffusion Models for Vision: A Surveycited by 51
  34. 2019ALIME: Autoencoder Based Approach for Local Interpretabilitycited by 41
  35. 2020Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologiescited by 41
  36. 2018Defining Locality for Surrogates in Post-hoc Interpretablitycited by 33
  37. 2023Text-Driven Image Editing via Learnable Regionscited by 31
  38. 2024Sora Generates Videos with Stunning Geometrical Consistencycited by 31

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Explaining Large Language Models with gSMILEcited by 4
  2. 2025Explainable Knowledge Graph Retrieval-Augmented Generation (KG-RAG) with KG-SMILEcited by 4
  3. 2026ConceptSMILE: Auditing the Trustworthiness of Concept-Based Explainable AIcited by 1