mlsanityData to 7 Oct 2026

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

arXiv 2407.18901 ↗First posted 26 Jul 2024cs.SE cs.AI cs.CLcited by 359

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Autonomous agents that address day-to-day digital tasks (e.g., ordering groceries for a household), must not only operate multiple apps (e.g., notes, messaging, shopping app) via APIs, but also generate rich code with complex control flow in an iterative manner based on their interaction with the environment. However, existing benchmarks for tool use are inadequate, as they only cover tasks that require a simple sequence of API calls. To remedy this gap, we built AppWorld Engine, a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs and populated with realistic digital activities simulating the lives of ~100 fictitious users. We then created AppWorld Benchmark (40K lines of code), a suite of 750 natural, diverse, and challenging autonomous agent tasks requiring rich and interactive code generation. It supports robust programmatic evaluation with state-based unit tests, allowing for different ways of completing a task while also checking for unexpected changes, i.e., collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our 'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least 16% fewer. This highlights the benchmark's difficulty and AppWorld's potential to push the frontiers of interactive coding agents. The project website is available at https://appworld.dev/.

Key facts. Cited by 359 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 3 of its 45 references that are in the map. First posted on arXiv 26 Jul 2024.

Builds on

3 of 45 references.

References · 3

  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  2. 2024MMInA: Benchmarking Multihop Multimodal Internet Agentscited by 55
  3. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 66; 2026: 282.

359 papers.

Sort:

By area:

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  4. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 349
  5. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  6. 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
  7. 2025Survey on Evaluation of LLM-based Agentscited by 190
  8. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  9. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
  10. 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 129
  11. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  12. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
38 more
  1. 2025Reinforcement Learning for Self-Improving Agent with Skill Librarycited by 91
  2. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 89
  3. 2025Agent Learning via Early Experiencecited by 79
  4. 2026Self-Harness: Harnesses That Improve Themselvescited by 79
  5. 2024Beyond Browsing: API-Based Web Agentscited by 77
  6. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
  7. 2025The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Executioncited by 73
  8. 2026SkillX: Automatically Constructing Skill Knowledge Bases for Agentscited by 69
  9. 2025Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolutioncited by 65
  10. 2025Self-Challenging Language Model Agentscited by 56
  11. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  12. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 43
  13. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  14. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  15. 2026AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agentscited by 32
  16. 2024Foundations and Recent Trends in Multimodal Mobile Agents: A Surveycited by 31
  17. 2025Towards Enterprise-Ready Computer Using Generalist Agentcited by 31
  18. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 31
  19. 2026PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agentscited by 30
  20. 2026WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluationcited by 29
  21. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  22. 2025ARE: Scaling Up Agent Environments and Evaluationscited by 28
  23. 2026Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environmentscited by 28
  24. 2026SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?cited by 28
  25. 2025PersonaBench: Evaluating AI Models on Understanding Personal Information through Accessing (Synthetic) Private User Datacited by 26
  26. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25
  27. 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Breakcited by 25
  28. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  29. 2025SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graphcited by 22
  30. 2026ReCreate: Reasoning and Creating Domain Agents Driven by Experiencecited by 22
  31. 2026From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agentscited by 22
  32. 2025Procedural Environment Generation for Tool-Use Agentscited by 21
  33. 2026From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flawscited by 21
  34. 2024FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answeringcited by 19
  35. 2025Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environmentscited by 19
  36. 2026SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agentscited by 19
  37. 2026Cordon: Semantic Transactions for Tool-Using LLM Agentscited by 19
  38. 2026SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolutioncited by 18

50 most cited of 359 All 359 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments
  2. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
  3. 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
  4. 2026VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents
  5. 2026SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
  6. 2026SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
  7. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
  8. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging