mlsanityData to 14 Oct 2026

Turning Intent into Specifications: A Benchmark and an Interactive User-Assistant Agent

arXiv 2606.20585 ↗First posted 13 May 2026cs.HC cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Today's agents are highly effective at implementing well-scoped software design plans, but user intent is often vague and admits multiple equally valid solutions. In this paper, we introduce SpecBench, a new benchmark for evaluating an agent's ability to translate user intent into a structured, executable specification that aligns with user preferences. The agent is given access to past user conversations and may interact with the user for a fixed number of rounds to ask clarifying questions. We find that existing agents exhibit two extreme behaviors: they either (i) struggle to collaborate proactively with users, entering implementation mode too quickly while overestimating their understanding of user preferences, or (ii) exhaust their question budget by asking about every ambiguous design choice. To address this limitation, we introduce a user-assistant agent: Buddy. It follows a workflow inspired by classical morphological analysis, decomposing user intent into a structured space of design dimensions and candidate choices. It then creates simulated users to evaluate these choices, before engaging the real user to resolve remaining ambiguities and finalize the specification. By shifting the focus from execution to specification, SpecBench and Buddy emphasize agent-user collaboration (not just code generation) as a key frontier in future agent design.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 47 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

41 of 47 references.

References · 41

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  3. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  6. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  8. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  10. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  11. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  12. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
29 more
  1. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  2. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  3. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  4. 2025Why Language Models Hallucinatecited by 220
  5. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  6. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
  7. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  8. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 106
  9. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 105
  10. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
  11. 2023Eliciting Human Preferences with Language Modelscited by 92
  12. 2017Learning how to Active Learn: A Deep Reinforcement Learning Approachcited by 79
  13. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 76
  14. 2018Interpretation of Natural Language Rules in Conversational Machine Readingcited by 75
  15. 2023On the Calibration of Large Language Models and Alignmentcited by 73
  16. 2025The Hallucination Tax of Reinforcement Finetuningcited by 40
  17. 2023ClarifyGPT: Empowering LLM-based Code Generation with Intention Clarificationcited by 38
  18. 2024Large Language Model-based Human-Agent Collaboration for Complex Task Solvingcited by 38
  19. 2025Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineeringcited by 38
  20. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  21. 2023A Survey on Asking Clarification Questions Datasets in Conversational Systemscited by 28
  22. 2025Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wildcited by 25
  23. 2025How Far are LLMs from Being Our Digital Twins? A Benchmark for Persona-Based Behavior Chain Simulationcited by 22
  24. 2025Know What You Don't Know: Uncertainty Calibration of Process Reward Modelscited by 21
  25. 2026Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agentscited by 19
  26. 2025InteractComp: Evaluating Search Agents With Ambiguous Queriescited by 12
  27. 2025Adaptive Elicitation of Latent Information Using Natural Languagecited by 7
  28. 2023Does Collaborative Human-LM Dialogue Generation Help Information Extraction from Human Dialogues?cited by 5
  29. 2024Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gaincited by 5

Built on by

Not cited yet.