# Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing arXiv: https://arxiv.org/abs/2506.16136 ยท HTML: https://mlsanity.com/paper/2506.16136/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. - Builds on 27 of its 99 references that are in the map. - First posted on arXiv 19 Jun 2025. ## Placed in - LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/) ## Builds on (most cited first, 27 of 27) - [SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution](https://mlsanity.com/paper/2502.18449/) (2025) - [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024) - [SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement](https://mlsanity.com/paper/2410.20285/) (2024) - [WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch](https://mlsanity.com/paper/2505.03733/) (2025) - [MarsCode Agent: AI-native Automated Bug Fixing](https://mlsanity.com/paper/2409.00899/) (2024) - [A Survey on (M)LLM-Based GUI Agents](https://mlsanity.com/paper/2504.13865/) (2025) - [A Survey on Large Language Models for Software Engineering](https://mlsanity.com/paper/2312.15223/) (2023) - [Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration](https://mlsanity.com/paper/2406.01422/) (2024) - [Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2502.10708/) (2025) - [RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair](https://mlsanity.com/paper/2312.15698/) (2023) - [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024) - [A Systematic Literature Review on Large Language Models for Automated Program Repair](https://mlsanity.com/paper/2405.01466/) (2024) - [UniDebugger: Hierarchical Multi-Agent Framework for Unified Software Debugging](https://mlsanity.com/paper/2404.17153/) (2024) - [Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach](https://mlsanity.com/paper/2406.16386/) (2024) - [An Empirical Study on LLM-based Agents for Automated Bug Fixing](https://mlsanity.com/paper/2411.10213/) (2024) - [Agentic Bug Reproduction for Effective Automated Program Repair at Google](https://mlsanity.com/paper/2502.01821/) (2025) - [ContrastRepair: Enhancing Conversation-Based Automated Program Repair via Contrastive Test Case Pairs](https://mlsanity.com/paper/2403.01971/) (2024) - [How Far Can We Go with Practical Function-Level Program Repair?](https://mlsanity.com/paper/2404.12833/) (2024) - [Aligning the Objective of LLM-based Program Repair](https://mlsanity.com/paper/2404.08877/) (2024) - [Doc2OracLL: Investigating the Impact of Documentation on LLM-based Test Oracle Generation](https://mlsanity.com/paper/2412.09360/) (2024) - [Vision-Based Mobile App GUI Testing: A Survey](https://mlsanity.com/paper/2310.13518/) (2023) - [DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models](https://mlsanity.com/paper/2411.01606/) (2024) - [PredicateFix: Repairing Static Analysis Alerts with Bridging Predicates](https://mlsanity.com/paper/2503.12205/) (2025) - [Programming with Pixels: Can Computer-Use Agents do Software Engineering?](https://mlsanity.com/paper/2502.18525/) (2025) - [Automated Soap Opera Testing Directed by LLMs and Scenario Knowledge: Feasibility, Challenges, and Road Ahead](https://mlsanity.com/paper/2412.08581/) (2024) - [Adversarial Reasoning for Repair Based on Inferred Program Intent](https://mlsanity.com/paper/2505.13008/) (2025) - [Fixing 7,400 Bugs for 1: Cheap Crash-Site Program Repair](https://mlsanity.com/paper/2505.13103/) (2025) ## Built on by (most cited first, 12 of 12) - [WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning](https://mlsanity.com/paper/2509.22644/) (2025) - [Agentic Software Issue Resolution with Large Language Models: A Survey](https://mlsanity.com/paper/2512.22256/) (2025) - [RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair](https://mlsanity.com/paper/2603.01048/) (2026) - [Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence](https://mlsanity.com/paper/2606.15932/) (2026) - [MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation](https://mlsanity.com/paper/2604.15309/) (2026) - [LLM Agents Can See Code Repositories](https://mlsanity.com/paper/2606.14061/) (2026) - [VulnResolver: A Hybrid Agent Framework for LLM-Based Automated Vulnerability Issue Resolution](https://mlsanity.com/paper/2601.13933/) (2026) - [SVRepair: Structured Visual Reasoning for Automated Program Repair](https://mlsanity.com/paper/2602.06090/) (2026) - [PassNet: Scaling Large Language Models for Graph Compiler Pass Generation](https://mlsanity.com/paper/2605.29357/) (2026) - [Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testing](https://mlsanity.com/paper/2607.01728/) (2026) - [Beyond Textual Repository Exploration: Dual-Modal Structural Reasoning for Agentic Issue Resolution](https://mlsanity.com/paper/2607.01929/) (2026) - [Lost in Perception: Isolating Perceptual and Reasoning Failures in Multimodal Physics and Geometry Reasoning](https://mlsanity.com/paper/2609.18991/) (2026) ## Abstract (arXiv) Large language model-(LLM) based automated program repair (APR) techniques have shown promising results in resolving real-world GitHub issue tasks. Existing APR systems are primarily evaluated in unimodal settings (e.g., SWE-bench). However, these autonomous systems struggle to resolve multimodal problem scenarios (e.g., SWE-bench M) due to limitations in interpreting and leveraging visual information. In multimodal scenarios, LLMs need to rely on visual information in the graphical user interface (GUI) to understand bugs and generate fixes. To bridge this gap, we propose GUIRepair, a cross-modal reasoning approach for resolving multimodal issue scenarios by understanding and capturing visual information. Specifically, GUIRepair integrates two key components, Image2Code and Code2Image, to enhance fault comprehension and patch validation. Image2Code extracts relevant project documents based on the issue report, then applies this domain knowledge to generate the reproduced code responsible for the visual symptoms, effectively translating GUI images into executable context for better fault comprehension. Code2Image replays the visual issue scenario using the reproduced code and captures GUI renderings of the patched program to assess whether the fix visually resolves the issue, providing feedback for patch validation. We evaluate GUIRepair on SWE-bench M, and the approach demonstrates significant effectiveness. When utilizing GPT-4o as the base model, GUIRepair solves 157 instances, outperforming the best open-source baseline by 26 instances. Furthermore, when using o4-mini as the base model, GUIRepair can achieve even better results and solve 175 instances, outperforming the top commercial system by 22 instances. This emphasizes the success of our new perspective on incorporating cross-modal reasoning by understanding and capturing visual information to resolve multimodal issues.