# FeatureBench: Benchmarking Agentic Coding for Complex Feature Development

arXiv: https://arxiv.org/abs/2602.10975 · HTML: https://mlsanity.com/paper/2602.10975/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 21 of 21)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents](https://mlsanity.com/paper/2504.07164/) (2025)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [FullStack Bench: Evaluating LLMs as Full Stack Coders](https://mlsanity.com/paper/2412.00535/) (2024)
- [Commit0: Library Generation from Scratch](https://mlsanity.com/paper/2412.01769/) (2024)
- [Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study](https://mlsanity.com/paper/2403.08604/) (2024)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunities](https://mlsanity.com/paper/2508.11126/) (2025)
- [GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging](https://mlsanity.com/paper/2508.18993/) (2025)
- [SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner](https://mlsanity.com/paper/2506.09003/) (2025)
- [SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development](https://mlsanity.com/paper/2505.16975/) (2025)
- [Language Models for Code Optimization: Survey, Challenges and Future Directions](https://mlsanity.com/paper/2501.01277/) (2025)

## Built on by (most cited first, 30 of 34)

- [SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks](https://mlsanity.com/paper/2603.24755/) (2026)
- [From Question Answering to Task Completion: A Survey on Agent System and Harness Design](https://mlsanity.com/paper/2606.20683/) (2026)
- [Agent psychometrics: Task-level performance prediction in agentic coding benchmarks](https://mlsanity.com/paper/2604.00594/) (2026)
- [Agent-as-a-Router: Agentic Model Routing for Coding Tasks](https://mlsanity.com/paper/2606.22902/) (2026)
- [SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution](https://mlsanity.com/paper/2603.13428/) (2026)
- [Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations](https://mlsanity.com/paper/2605.11633/) (2026)
- [SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle](https://mlsanity.com/paper/2605.13139/) (2026)
- [Code Review Agent Benchmark](https://mlsanity.com/paper/2603.23448/) (2026)
- [Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation in an Uncertain Enterprise Environment](https://mlsanity.com/paper/2603.23638/) (2026)
- [RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades](https://mlsanity.com/paper/2605.15846/) (2026)
- [Same Model, Different Harness: Different Coding-Agent Results](https://mlsanity.com/paper/2608.26218/) (2026)
- [ATM: CID-Brokered Pre-Write Admission for Multi-Agent Code Co-Synthesis](https://mlsanity.com/paper/2607.00041/) (2026)
- [LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation](https://mlsanity.com/paper/2608.00267/) (2026)
- [CodeSpec: Dual Executable Specifications for Agentic Long-Horizon Feature Development](https://mlsanity.com/paper/2607.26777/) (2026)
- [MindForge: Teaching Small Language Models Whole-Life-Cycle Software Engineering via Source-Free Program Synthesis](https://mlsanity.com/paper/2607.27146/) (2026)
- [Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents](https://mlsanity.com/paper/2605.21347/) (2026)
- [Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World](https://mlsanity.com/paper/2605.26086/) (2026)
- [Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments](https://mlsanity.com/paper/2607.28591/) (2026)
- [Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports](https://mlsanity.com/paper/2608.04682/) (2026)
- [Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation](https://mlsanity.com/paper/2609.04298/) (2026)
- [LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems](https://mlsanity.com/paper/2609.37143/) (2026)
- [Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents](https://mlsanity.com/paper/2609.39982/) (2026)
- [Scaling Human-AI Coding Collaboration Requires a Governable Consensus Layer](https://mlsanity.com/paper/2604.17883/) (2026)
- [Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations](https://mlsanity.com/paper/2607.02684/) (2026)
- [PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization](https://mlsanity.com/paper/2607.07744/) (2026)
- [BulkPR-Bench: Benchmarking Queue-Level Governance of Interacting Pull Requests](https://mlsanity.com/paper/2608.02685/) (2026)
- [Scaffold-Mediated Post-Training: Co-Evolving Model Parameters and Procedural Scaffold Graphs](https://mlsanity.com/paper/2608.05156/) (2026)
- [SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving](https://mlsanity.com/paper/2609.26777/) (2026)
- [Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding Agents](https://mlsanity.com/paper/2610.06193/) (2026)
- [SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction](https://mlsanity.com/paper/2610.11559/) (2026)

## Abstract (arXiv)

Agents powered by large language models (LLMs) are increasingly adopted in the software industry, contributing code as collaborators or even autonomous developers. As their presence grows, it becomes important to assess the current boundaries of their coding abilities. Existing agentic coding benchmarks, however, cover a limited task scope, e.g., bug fixing within a single pull request (PR), and often rely on non-executable evaluations or lack an automated approach for continually updating the evaluation coverage. To address such issues, we propose FeatureBench, a benchmark designed to evaluate agentic coding performance in end-to-end, feature-oriented software development. FeatureBench incorporates an execution-based evaluation protocol and a scalable test-driven method that automatically derives tasks from code repositories with minimal human effort. By tracing from unit tests along a dependency graph, our approach can identify feature-level coding tasks spanning multiple commits and PRs scattered across the development timeline, while ensuring the proper functioning of other features after the separation. Using this framework, we curated 200 challenging evaluation tasks and 3825 executable environments from 24 open-source repositories in the first version of our benchmark. Empirical evaluation reveals that the state-of-the-art agentic model, such as Claude 4.5 Opus, which achieves a 74.4% resolved rate on SWE-bench, succeeds on only 11.0% of tasks, opening new opportunities for advancing agentic coding. Moreover, benefiting from our automated task collection toolkit, FeatureBench can be easily scaled and updated over time to mitigate data leakage. The inherent verifiability of constructed environments also makes our method potentially valuable for agent training.