# ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development

arXiv: https://arxiv.org/abs/2601.11077 · HTML: https://mlsanity.com/paper/2601.11077/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 16 Jan 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 30)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [SWE-Bench+: Enhanced Coding Benchmark for LLMs](https://mlsanity.com/paper/2410.06992/) (2024)
- [Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?](https://mlsanity.com/paper/2511.13646/) (2025)
- [Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling](https://mlsanity.com/paper/2507.23370/) (2025)
- [SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents](https://mlsanity.com/paper/2508.02085/) (2025)
- [FullStack Bench: Evaluating LLMs as Full Stack Coders](https://mlsanity.com/paper/2412.00535/) (2024)
- [FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation](https://mlsanity.com/paper/2503.06680/) (2025)
- [UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench](https://mlsanity.com/paper/2506.09289/) (2025)
- [Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study](https://mlsanity.com/paper/2403.08604/) (2024)
- [McEval: Massively Multilingual Code Evaluation](https://mlsanity.com/paper/2406.07436/) (2024)
- [SWE-bench-java: A GitHub Issue Resolving Benchmark for Java](https://mlsanity.com/paper/2408.14354/) (2024)
- [PyBench: Evaluating LLM Agent on various real-world coding tasks](https://mlsanity.com/paper/2407.16732/) (2024)
- [CodeEditorBench: Evaluating Code Editing Capability of Large Language Models](https://mlsanity.com/paper/2404.03543/) (2024)
- [ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows](https://mlsanity.com/paper/2502.00964/) (2025)
- [Nex-N1: Agentic Models Trained via a Unified Ecosystem for Large-Scale Environment Construction](https://mlsanity.com/paper/2512.04987/) (2025)
- [Code Needs Comments: Enhancing Code LLMs with Comment Augmentation](https://mlsanity.com/paper/2402.13013/) (2024)
- [Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation](https://mlsanity.com/paper/2510.26130/) (2025)

## Built on by (most cited first, 6 of 6)

- [From Question Answering to Task Completion: A Survey on Agent System and Harness Design](https://mlsanity.com/paper/2606.20683/) (2026)
- [MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents](https://mlsanity.com/paper/2605.03952/) (2026)
- [Execution-First Synthetic Tool-Use Trace Generation for LLM Agents](https://mlsanity.com/paper/2607.29175/) (2026)
- [Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation](https://mlsanity.com/paper/2609.04298/) (2026)
- [BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services](https://mlsanity.com/paper/2607.11042/) (2026)
- [Atria Dawn: The Dawn of Agentic Superintelligence](https://mlsanity.com/paper/2609.15818/) (2026)

## Abstract (arXiv)

The evolution of Large Language Models (LLMs) into autonomous agents has expanded the scope of AI coding from localized code generation to complex, repository-level, and execution-driven problem solving. However, current benchmarks predominantly evaluate code logic in static contexts, neglecting the dynamic, full-process requirements of real-world engineering, particularly in backend development which demands rigorous environment configuration and service deployment. To address this gap, we introduce ABC-Bench, a benchmark explicitly designed to evaluate agentic backend coding within a realistic, executable workflow. Using a scalable automated pipeline, we curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories. Distinct from previous evaluations, ABC-Bench require the agents to manage the entire development lifecycle from repository exploration to instantiating containerized services and pass the external end-to-end API tests. Our extensive evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks, highlighting a substantial disparity between current model capabilities and the demands of practical backend engineering. Our code is available at https://github.com/OpenMOSS/ABC-Bench.