# EditScribe: Non-Visual Image Editing with Natural Language Verification Loops

arXiv: https://arxiv.org/abs/2408.06632 · HTML: https://mlsanity.com/paper/2408.06632/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 89 references that are in the map.
- First posted on arXiv 13 Aug 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://mlsanity.com/paper/2311.10089/) (2023)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Inpaint Anything: Segment Anything Meets Image Inpainting](https://mlsanity.com/paper/2304.06790/) (2023)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [Inst-Inpaint: Instructing to Remove Objects with Diffusion Models](https://mlsanity.com/paper/2304.03246/) (2023)
- [UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Image](https://mlsanity.com/paper/2210.09477/) (2022)
- [ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing](https://mlsanity.com/paper/2303.02437/) (2023)
- [AVscript: Accessible Video Editing with Audio-Visual Scripts](https://mlsanity.com/paper/2302.14117/) (2023)
- [GenAssist: Making Image Generation Accessible](https://mlsanity.com/paper/2307.07589/) (2023)
- [Making Short-Form Videos Accessible with Hierarchical Video Summaries](https://mlsanity.com/paper/2402.10382/) (2024)
- [The Effect of Computer-Generated Descriptions on Photo-Sharing Experiences of People with Visual Impairments](https://mlsanity.com/paper/1805.01515/) (2018)
- ["It Feels Like Being Locked in A Cage": Understanding Blind or Low Vision Streamers' Perceptions of Content Curation Algorithms](https://mlsanity.com/paper/2204.11247/) (2022)

## Built on by (most cited first, 6 of 6)

- [Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions](https://mlsanity.com/paper/2507.15692/) (2025)
- [Substance Beats Style: Why Beginning Students Fail to Code with LLMs](https://mlsanity.com/paper/2410.19792/) (2024)
- ["It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models](https://mlsanity.com/paper/2511.08917/) (2025)
- [Probing the Gaps in ChatGPT Live Video Chat for Real-World Assistance for People who are Blind or Visually Impaired](https://mlsanity.com/paper/2508.03651/) (2025)
- [CoEditor++: Instruction-based Visual Editing via Cognitive Reasoning](https://mlsanity.com/paper/2603.05518/) (2026)
- [Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents](https://mlsanity.com/paper/2607.17527/) (2026)

## Abstract (arXiv)

Image editing is an iterative process that requires precise visual evaluation and manipulation for the output to match the editing intent. However, current image editing tools do not provide accessible interaction nor sufficient feedback for blind and low vision individuals to achieve this level of control. To address this, we developed EditScribe, a prototype system that makes image editing accessible using natural language verification loops powered by large multimodal models. Using EditScribe, the user first comprehends the image content through initial general and object descriptions, then specifies edit actions using open-ended natural language prompts. EditScribe performs the image edit, and provides four types of verification feedback for the user to verify the performed edit, including a summary of visual changes, AI judgement, and updated general and object descriptions. The user can ask follow-up questions to clarify and probe into the edits or verification feedback, before performing another edit. In a study with ten blind or low-vision users, we found that EditScribe supported participants to perform and verify image edit actions non-visually. We observed different prompting strategies from participants, and their perceptions on the various types of verification feedback. Finally, we discuss the implications of leveraging natural language verification loops to make visual authoring non-visually accessible.