mlsanityData to 7 Oct 2026

Asuka-Bench: Benchmarking Code Agents on Underspecified User Intent and Multi-Round Refinement

arXiv 2606.05920 ↗First posted 4 Jun 2026cs.SE cs.CLcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Existing code-generation benchmarks score a single mapping from a complete prompt to a one-shot output. However, real web development is different. Users seldom write a full spec at the start; many requirements only become clear once they look at an intermediate result and react to it. We present Asuka-Bench, a benchmark that pairs underspecified user intent with multi-round refinement, grounded in browser-rendered behavior. Each task is resolved through a closed loop: a Code Agent generates a web project, a UI Agent executes test cases on the deployed site, and a User LLM turns evaluation outcomes into natural-language feedback for the next round. The benchmark comprises 50 web tasks with 784 evaluation criteria and 2402 expected outcomes. We benchmark 8 LLMs across 2 agent frameworks. The results separate models clearly: weighted Task Pass Rate varies by 38 percentage points and models also differ substantially in their ability to repair from feedback. Asuka-Bench is also far from saturated: even the strongest model completes only 52% of projects after three rounds.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 24 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

12 of 24 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation28 Oct
  2. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks12 May
  3. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch6 May

Other references · 9

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  6. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
  7. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluationcited by 35
  8. 2025WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generationcited by 16
  9. 2025WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in WebUI-to-Codecited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessionscited by 8
  2. 2026What Do We Expect from LLMs? Mapping the Design of LLM Benchmarkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics
  2. 2026VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents
  3. 2026ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
  4. 2026Coding-Agent Benchmarks Should Match Their Users' Task Flows
  5. 2026SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
  6. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
  7. 2026WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing
  8. 2026LoLBench: Evaluating Coding Agents with Long-Horizon Proposals on Large Software Systems