mlsanityData to 14 Oct 2026

SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?

arXiv 2410.03859 ↗First posted 4 Oct 2024cs.CL cs.AI cs.SEcited by 203

Not on the map. Why?

Cited by 121 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual elements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains (e.g., front-end, game development, DevOps), which use different programming languages and paradigms. Therefore, we propose SWE-bench Multimodal (SWE-bench M), to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software. SWE-bench M features 617 task instances collected from 17 JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Each SWE-bench M task instance contains at least one image in its problem statement or unit tests. Our analysis finds that top-performing SWE-bench systems struggle with SWE-bench M, revealing limitations in visual problem-solving and cross-language generalization. Lastly, we show that SWE-agent's flexible language-agnostic features enable it to substantially outperform alternatives on SWE-bench M, resolving 12% of task instances compared to 6% for the next best system.

Key facts. Cited by 203 papers in the map (data through 14 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

34 of 57 references.

References · 34

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2017Billion-scale similarity search with GPUscited by 2,497
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  10. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  12. 2021Measuring Coding Challenge Competence With APPScited by 902
22 more
  1. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  3. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  4. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  5. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  6. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  7. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  8. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 238
  9. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  10. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  11. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  12. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 114
  13. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  14. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87
  15. 2024Evaluating Language Models for Efficient Code Generationcited by 73
  16. 2024EffiBench: Benchmarking the Efficiency of Automatically Generated Codecited by 64
  17. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilitiescited by 52
  18. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 48
  19. 2022MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languagescited by 41
  20. 2024ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?cited by 27
  21. 2024SVGEditBench: A Benchmark Dataset for Quantitative Assessment of LLM's SVG Editing Capabilitiescited by 26
  22. 2019Sketch2code: Generating a website from a paper mockupcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 82; 2026: 118.

203 papers.

Sort:

By area:

  1. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  2. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  3. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  4. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 225
  5. 2025Survey on Evaluation of LLM-based Agentscited by 204
  6. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
  7. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  8. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 142
  9. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
  10. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  11. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 85
  12. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 72
38 more
  1. 2025AutoPresent: Designing Structured Visuals from Scratchcited by 68
  2. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 64
  3. 2025LLM Agents for Education: Advances and Applicationscited by 59
  4. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 59
  5. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 59
  6. 2024Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototypingcited by 51
  7. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 50
  8. 2026DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Taskscited by 49
  9. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  10. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 46
  11. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generationcited by 45
  12. 2026SWE-chat: Coding Agent Interactions From Real Users in the Wildcited by 43
  13. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  14. 2024TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?cited by 34
  15. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfacescited by 34
  16. 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 34
  17. 2026SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolvingcited by 31
  18. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  19. 2026GameDevBench: Evaluating Agentic Capabilities Through Game Developmentcited by 28
  20. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 26
  21. 2025GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agentscited by 25
  22. 2025Coding Agents with Multimodal Browsing are Generalist Problem Solverscited by 25
  23. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  24. 2025TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understandingcited by 24
  25. 2025Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approachescited by 23
  26. 2026AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarioscited by 23
  27. 2026SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integrationcited by 23
  28. 2025Agentic Bug Reproduction for Effective Automated Program Repair at Googlecited by 22
  29. 2025Issue Localization via LLM-Driven Iterative Code Graph Searchingcited by 22
  30. 2025Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Taskscited by 22
  31. 2026SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Trainingcited by 22
  32. 2026Immersion in the GitHub Universe: Scaling Coding Agents to Masterycited by 22
  33. 2026BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?cited by 22
  34. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  35. 2025Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports From Scratch with Agentic Frameworkcited by 21
  36. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 21
  37. 2026SWE-Explore: Benchmarking How Coding Agents Explore Repositoriescited by 21
  38. 2025Imitation Learning for Multi-turn LM Agents via On-policy Expert Correctionscited by 19

50 most cited of 203 All 203 (JSON)