# AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineering

arXiv: https://arxiv.org/abs/2609.32662 · HTML: https://mlsanity.com/paper/2609.32662/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 30 of 36)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks](https://mlsanity.com/paper/2411.04468/) (2024)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion](https://mlsanity.com/paper/2310.11248/) (2023)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents](https://mlsanity.com/paper/2505.20411/) (2025)
- [SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?](https://mlsanity.com/paper/2502.12115/) (2025)
- [SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents](https://mlsanity.com/paper/2504.08703/) (2025)
- [NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents](https://mlsanity.com/paper/2512.12730/) (2025)
- [SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios](https://mlsanity.com/paper/2512.18470/) (2025)
- [FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation](https://mlsanity.com/paper/2503.06680/) (2025)
- [RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code](https://mlsanity.com/paper/2503.07832/) (2025)
- [Automated Benchmark Generation for Repository-Level Coding Tasks](https://mlsanity.com/paper/2503.07701/) (2025)
- [SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale](https://mlsanity.com/paper/2602.23866/) (2026)
- [CooperBench: Why Coding Agents Cannot be Your Teammates Yet](https://mlsanity.com/paper/2601.13295/) (2026)
- [Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems](https://mlsanity.com/paper/2603.01045/) (2026)
- [SWE-Together: Evaluating Coding Agents in Interactive User Sessions](https://mlsanity.com/paper/2606.29957/) (2026)
- [Effective Strategies for Asynchronous Software Engineering Agents](https://mlsanity.com/paper/2603.21489/) (2026)
- [Are Tools All We Need? Unveiling the Tool-Use Tax in LLM Agents](https://mlsanity.com/paper/2605.00136/) (2026)
- [TeamBench: Evaluating Agent Coordination under Enforced Role Separation](https://mlsanity.com/paper/2605.07073/) (2026)
- [SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering](https://mlsanity.com/paper/2502.06994/) (2025)
- [AgentSpawn: Adaptive Multi-Agent Collaboration Through Dynamic Spawning for Long-Horizon Code Generation](https://mlsanity.com/paper/2602.07072/) (2026)
- [AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering](https://mlsanity.com/paper/2604.13120/) (2026)

## Abstract (arXiv)

Multi-agent coding has emerged as an increasingly active direction in software engineering, where complex development tasks are decomposed across multiple specialized agents working on different parts of the problem. Despite the shift from individual problem solving to distributed collaboration, multi-agent systems still lack a direct measure of collaboration and are largely evaluated through task-level outcomes inherited from single-agent coding, conflating individual coding capability with cross-agent coordination. We introduce AsynCodeBench, a dependency-centric benchmark for asynchronous multi-agent software engineering that represents each task with an explicit dependency graph and executable Dependency Checkers. Through this dependency-tracking process, we propose two complementary measures: Asynchronous Dependency Pass Rate (ADPR), which measures how many cross-agent dependencies are ultimately satisfied, and Dependency Resolution Step (DRS), which measures when each dependency first becomes satisfied during execution. AsynCodeBench comprises 19 tasks from real-world repositories, exposing 52 directed dependencies as explicit units for evaluating cross-agent collaboration. Experiments across model families, scales, and generations reveal a clear gap between coding and collaboration capability: improvements in coding performance do not necessarily translate into stronger collaboration, and task-level metrics can diverge substantially from dependency-level collaboration measures. Dependency-trajectory analysis further reveals that successful coordination often emerges not gradually, but through concentrated bursts in which many dependencies become resolved over a short portion of the execution trajectory, a pattern we term a hopping window.