# Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modeling arXiv: https://arxiv.org/abs/2511.03404 ยท HTML: https://mlsanity.com/paper/2511.03404/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 21 of its 28 references that are in the map. - First posted on arXiv 5 Nov 2025. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 21 of 21) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024) - [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024) - [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023) - [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020) - [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024) - [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023) - [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024) - [CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion](https://mlsanity.com/paper/2310.11248/) (2023) - [DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories](https://mlsanity.com/paper/2405.19856/) (2024) - [Repoformer: Selective Retrieval for Repository-Level Code Completion](https://mlsanity.com/paper/2403.10059/) (2024) - [Commit0: Library Generation from Scratch](https://mlsanity.com/paper/2412.01769/) (2024) - [Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback](https://mlsanity.com/paper/2403.16792/) (2024) - [AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodology](https://mlsanity.com/paper/2406.11912/) (2024) - [Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study](https://mlsanity.com/paper/2403.08604/) (2024) - [CodeS: Natural Language to Code Repository via Multi-Layer Sketch](https://mlsanity.com/paper/2403.16443/) (2024) - [ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation](https://mlsanity.com/paper/2503.07010/) (2025) - [Experimenting a New Programming Practice with LLMs](https://mlsanity.com/paper/2401.01062/) (2024) - [SRLCG: Self-Rectified Large-Scale Code Generation with Multidimensional Chain-of-Thought and Dynamic Backtracking](https://mlsanity.com/paper/2504.00532/) (2025) ## Built on by (most cited first, 6 of 6) - [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025) - [When Parallelism Pays Off: Cohesion-Aware Task Partitioning for Multi-Agent Coding](https://mlsanity.com/paper/2606.00953/) (2026) - [CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation](https://mlsanity.com/paper/2606.22082/) (2026) - [Do Papers Tell the Whole Story? A Benchmark and Framework for Uncovering Hidden Implementation Gaps in Bioinformatics](https://mlsanity.com/paper/2603.22018/) (2026) - [Contract-Coding: Towards Repo-Level Generation via Structured Symbolic Paradigm](https://mlsanity.com/paper/2604.13100/) (2026) - [CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language](https://mlsanity.com/paper/2603.14501/) (2026) ## Abstract (arXiv) In recent years, Large Language Models (LLMs) have achieved remarkable progress in automated code generation. In real-world software engineering, the growing demand for rapid iteration and continuous delivery underscores the importance of project-level code generation, where LLMs are expected to generate complete software projects directly from complex user requirements. Although existing studies have made initial explorations, they still face key limitations, including unrealistic datasets and unreliable evaluation metrics that fail to reflect real-world complexity, the semantic gap between human-written requirements and machine-interpretable structures, and difficulties in managing hierarchical dependencies and maintaining quality throughout the generation process. To address these limitations, we first introduce CodeProjectEval, a project-level code generation dataset built from 18 real-world repositories with 12.7 files and 2,388.6 lines of code per task on average, supplemented with documentation and executable test cases for automatic evaluation. We further propose ProjectGen, a multi-agent framework that decomposes projects into architecture design, skeleton generation, and code filling stages with iterative refinement and memory-based context management. Within this framework, we introduce the Semantic Software Architecture Tree (SSAT), a structured and semantically rich representation that effectively bridges user requirements and source code implementation. Experiments show that ProjectGen achieves state-of-the-art performance, passing 52/124 test cases on the small-scale project-level code generation dataset DevBench, a 57% improvement over the baseline approaches, and 310 test cases on CodeProjectEval, representing an improvement of roughly tenfold compared to the baselines.