# LLM Agents for Generating Microservice-based Applications: how complex is your specification?

arXiv: https://arxiv.org/abs/2508.20119 · HTML: https://mlsanity.com/paper/2508.20119/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Agentless: Demystifying LLM-based Software Engineering Agents](https://mlsanity.com/paper/2407.01489/) (2024)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems](https://mlsanity.com/paper/2306.03091/) (2023)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories](https://mlsanity.com/paper/2405.19856/) (2024)
- [We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs](https://mlsanity.com/paper/2406.10279/) (2024)
- [FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation](https://mlsanity.com/paper/2503.06680/) (2025)
- [What's Wrong with Your Code Generated by Large Language Models? An Extensive Study](https://mlsanity.com/paper/2407.06153/) (2024)
- [Type-Constrained Code Generation with Language Models](https://mlsanity.com/paper/2504.09246/) (2025)
- [Do Advanced Language Models Eliminate the Need for Prompt Engineering in Software Engineering?](https://mlsanity.com/paper/2411.02093/) (2024)
- [Harnessing LLMs for API Interactions: A Framework for Classification and Synthetic Data Generation](https://mlsanity.com/paper/2409.11703/) (2024)
- [GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models](https://mlsanity.com/paper/2411.05830/) (2024)
- [Control Large Language Models via Divide and Conquer](https://mlsanity.com/paper/2410.04628/) (2024)
- [AutoFeedback: An LLM-based Framework for Efficient and Accurate API Request Generation](https://mlsanity.com/paper/2410.06943/) (2024)
- [How Robust are LLM-Generated Library Imports? An Empirical Study using Stack Overflow](https://mlsanity.com/paper/2507.10818/) (2025)

## Abstract (arXiv)

In this paper we evaluate the capabilities of LLM Agents in generating code for real-world problems. Specifically, we explore code synthesis for microservice-based applications, a widely used architectural pattern for building applications. We define a standard template for specifying these applications, and we propose a metric for scoring the difficulty of a specification. The higher the score, the more difficult it is to generate code for the specification. Our experimental results show that agents using strong LLMs (like GPT-3o-mini) do fairly well on medium difficulty specifications but do poorly on those of higher difficulty levels. This is due to more intricate business logic, a greater use of external services, database integration and inclusion of non-functional capabilities such as authentication. We analyzed the errors in LLM-synthesized code and report on the key challenges LLM Agents face in generating code for these specifications. Finally, we show that using a fine-grained approach to code generation improves the correctness of the generated code.