mlsanityData to 14 Oct 2026

FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation

arXiv 2503.06680 ↗First posted 9 Mar 2025cs.SE cs.CLcited by 58

Not on the map. Why?

Cited by 35 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Implementing new features in repository-level codebases is a crucial application of code generation models. However, current benchmarks lack a dedicated evaluation framework for this capability. To fill this gap, we introduce FEA-Bench, a benchmark designed to assess the ability of large language models (LLMs) to perform incremental development within code repositories. We collect pull requests from 83 GitHub repositories and use rule-based and intent-based filtering to construct task instances focused on new feature development. Each task instance containing code changes is paired with relevant unit test files to ensure that the solution can be verified. The feature implementation requires LLMs to simultaneously possess code completion capabilities for new components and code editing abilities for other relevant parts in the code repository, providing a more comprehensive evaluation method of LLMs' automated software engineering capabilities. Experimental results show that LLMs perform significantly worse in the FEA-Bench, highlighting considerable challenges in such repository-level incremental code development.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 29 of its 43 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

29 of 43 references.

References · 29

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  11. 2021Measuring Coding Challenge Competence With APPScited by 902
  12. 2023StarCoder: may the source be with you!cited by 799
17 more
  1. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  2. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  3. 2022A Systematic Evaluation of Large Language Models of Codecited by 363
  4. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  5. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  6. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  7. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  8. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  9. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  10. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  11. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  12. 2022Repair Is Nearly Generation: Multilingual Program Repair with LLMscited by 68
  13. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
  14. 2023WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuningcited by 42
  15. 2024ExecRepoBench: Multi-level Executable Code Completion Evaluationcited by 20
  16. 2025EpiCoder: Encompassing Diversity and Complexity in Code Generationcited by 20
  17. 2024Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?cited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 44.

58 papers.

Sort:

By area:

  1. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfacescited by 34
  2. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskscited by 30
  3. 2025When Agents go Astray: Course-Correcting SWE Agents with PRMscited by 28
  4. 2025Issue Localization via LLM-Driven Iterative Code Graph Searchingcited by 22
  5. 2026Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?cited by 20
  6. 2025RPG: A Repository Planning Graph for Unified and Scalable Codebase Generationcited by 19
  7. 2026Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learnedcited by 18
  8. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  9. 2025A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Codecited by 14
  10. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarioscited by 13
  11. 2026Structurally Aligned Subtask-Level Memory for Software Engineering Agentscited by 12
  12. 2025Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agentscited by 11
38 more
  1. 2026CodeTaste: Can LLMs Generate Human-Level Code Refactorings?cited by 8
  2. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  3. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
  4. 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Developmentcited by 6
  5. 2026When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agentscited by 6
  6. 2025CFDLLMBench: A Benchmark Suite for Evaluating Large Language Models in Computational Fluid Dynamicscited by 5
  7. 2026TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generationcited by 5
  8. 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cyclecited by 5
  9. 2026MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineeringcited by 4
  10. 2026Code Review Agent Benchmarkcited by 4
  11. 2026CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generationcited by 4
  12. 2025CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Taskscited by 3
  13. 2026Learning to Commit: Generating Organic Pull Requests via Online Repository Memorycited by 3
  14. 2026Constraint Decay: The Fragility of LLM Agents in Backend Code Generationcited by 3
  15. 2025TENET: One Step Toward Test-Driven Development for Repository-Level Code Generationcited by 2
  16. 2025Generating High-Quality Datasets for Code Editing via Open-Source Language Modelscited by 2
  17. 2025CIFE: Code Instruction-Following Evaluationcited by 2
  18. 2026Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolutioncited by 2
  19. 2026PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimizationcited by 2
  20. 2026CodeSpec: Dual Executable Specifications for Agentic Long-Horizon Feature Developmentcited by 2
  21. 2026A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agentscited by 2
  22. 2026Specification-Driven Generation and Evaluation of Discrete-Event World Models via the DEVS Formalismcited by 1
  23. 2026Persistent Cross-Attempt State Optimization for Repository-Level Code Generationcited by 1
  24. 2026Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versionscited by 1
  25. 2026SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvementscited by 1
  26. 2026Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environmentscited by 1
  27. 2026GameASG-Bench: Benchmarking Autonomous Software Generation for Game Developmentcited by 1
  28. 2026LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systemscited by 1
  29. 2025LLM Agents for Generating Microservice-based Applications: how complex is your specification?cited by 0
  30. 2026SWE-Spot: Building Small Repo-Experts with Repository-Centric Learningcited by 0
  31. 2026Toward Executable Repository-Level Code Generation via Environment Alignmentcited by 0
  32. 2026SW-A2-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Webcited by 0
  33. 2026Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systemscited by 0
  34. 2026Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalencecited by 0
  35. 2026Vibe Coding Ate My Homework: An evaluation of AI approaches to greenfield software engineering and programmingcited by 0
  36. 2026SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agentscited by 0
  37. 2026LLVM-Bench: Benchmarking and Advancing Large Language Models for LLVM Compiler Issue Resolutioncited by 0
  38. 2026Rethinking Code Performance Benchmarks for LLMscited by 0

50 most cited of 58 All 58 (JSON)