mlsanityData to 14 Oct 2026

AppForge: From Assistant to Independent Developer -- Are GPTs Ready for Software Development?

arXiv 2510.07740 ↗First posted 9 Oct 2025cs.SE cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capability in function-level code generation tasks. Unlike isolated functions, real-world applications demand reasoning over the entire software system: developers must orchestrate how different components interact, maintain consistency across states over time, and ensure the application behaves correctly within the lifecycle and framework constraints. Yet, no existing benchmark adequately evaluates whether LLMs can bridge this gap and construct entire software systems from scratch. To address this gap, we propose APPFORGE, a benchmark consisting of 101 software development problems drawn from real-world Android apps. Given a natural language specification detailing the app functionality, a language model is tasked with implementing the functionality into an Android app from scratch. Developing an Android app from scratch requires understanding and coordinating app states, lifecycle management, and asynchronous operations, calling for LLMs to generate context-aware, robust, and maintainable code. To construct APPFORGE, we design a multi-agent system to automatically summarize the main functionalities from app documents and navigate the app to synthesize test cases validating the functional correctness of app implementation. Following rigorous manual verification by Android development experts, APPFORGE incorporates the test cases within an automated evaluation framework that enables reproducible assessment without human intervention, making it easily adoptable for future research. Our evaluation on 12 flagship LLMs show that all evaluated models achieve low effectiveness, with the best-performing model (GPT-5) developing only 18.8% functionally correct applications, highlighting fundamental limitations in current models' ability to handle complex, multi-component software engineering challenges.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

33 of 51 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  8. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  11. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  12. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
21 more
  1. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  2. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  3. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  4. 2023Self-planning Code Generation with Large Language Modelscited by 124
  5. 2023InferFix: End-to-End Program Repair with LLMscited by 114
  6. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 103
  7. 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 96
  8. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
  9. 2025SWE-bench Goes Live!cited by 80
  10. 2024Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLMcited by 64
  11. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworkscited by 40
  12. 2024TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMscited by 24
  13. 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contaminationcited by 23
  14. 2024STALL+: Boosting LLM-based Repository-level Code Completion with Static Analysiscited by 21
  15. 2024Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluationcited by 19
  16. 2025LAMD: Context-driven Android Malware Detection and Classification with LLMscited by 18
  17. 2023CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Modelcited by 14
  18. 2024KGym: A Platform and Dataset to Benchmark Large Language Models on Linux Kernel Crash Resolutioncited by 12
  19. 2025LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineeringcited by 12
  20. 2024PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Modelscited by 8
  21. 2025Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 18
  2. 2026On the Adoption of AI Coding Agents in Open-source Android and iOS Developmentcited by 1
  3. 2026Compiling Large Multi-Modal Requirement Documents into Runnable Software Systems: From an Agentic Test-Driven Perspectivecited by 1
  4. 2026RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agentscited by 1