# SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?

arXiv: https://arxiv.org/abs/2602.09540 · HTML: https://mlsanity.com/paper/2602.09540/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 12 of its 29 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 12 of 12)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation](https://mlsanity.com/paper/2308.01861/) (2023)
- [Execution-Based Evaluation for Open-Domain Code Generation](https://mlsanity.com/paper/2212.10481/) (2022)

## Built on by (most cited first, 12 of 12)

- [Meta-Harness: End-to-End Optimization of Model Harnesses](https://mlsanity.com/paper/2603.28052/) (2026)
- [SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment](https://mlsanity.com/paper/2604.13630/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection](https://mlsanity.com/paper/2608.20169/) (2026)
- [Towards Direct Evaluation of Harness Optimizers via Priority Ranking](https://mlsanity.com/paper/2605.22505/) (2026)
- [MILO: Automated Harness Discovery via Orchestrated Multi-Agent Evolution](https://mlsanity.com/paper/2609.38349/) (2026)
- [Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems](https://mlsanity.com/paper/2605.27492/) (2026)
- [Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?](https://mlsanity.com/paper/2606.19388/) (2026)
- [Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation](https://mlsanity.com/paper/2607.28645/) (2026)
- [Turbo Harness: Instance-Adaptive Harness Optimization](https://mlsanity.com/paper/2609.40330/) (2026)
- [HASTE: Evolving Agent Harnesses Against Emerging Attacks Using Sparse Evidence](https://mlsanity.com/paper/2610.02920/) (2026)
- [AIProver: Agentic Auto-Formalization of Mathematical Research via Certificate-Driven Evolving Harness](https://mlsanity.com/paper/2610.05367/) (2026)

## Abstract (arXiv)

Can large language model agents develop industry-level mobile applications? We introduce SWE-Bench Mobile, a benchmark for evaluating coding agents on realistic software engineering tasks derived from a production iOS codebase. Unlike existing benchmarks that focus on isolated problems or bug fixes, SWE-Bench Mobile captures the full complexity of industrial development: multi-modal inputs (PRDs and Figma designs), a large-scale mixed Swift/Objective-C codebase, and comprehensive test suites. We evaluate 22 agent-model configurations across four coding agents -- three commercial (Cursor, Codex, Claude Code) and one open-source (OpenCode) -- and find that even the best configurations achieve only 12% task success rate. Our analysis reveals that (1) agent design matters as much as model capability -- the same model shows up to 6× performance gap across agents, (2) commercial agents consistently outperform open-source alternatives, and (3) simple ``Defensive Programming'' prompts outperform complex ones by 7.4%. These findings highlight a significant gap between current agent capabilities and industrial requirements, while providing actionable insights for practitioners and researchers. We release SWE-Bench Mobile as a textithosted benchmark challenge to prevent data contamination and ensure fair evaluation. The public leaderboard and development toolkit are available at https://swebenchmobile.com.