mlsanityData to 14 Oct 2026

Measuring and Reducing Unusable Replies of LLM Agents

arXiv 2610.19659 ↗First posted 14 Oct 2026cs.CL cs.AI cs.HCcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate also fits: Web, search, and research agents

Abstract (arXiv)

People delegate multi-step work to LLM agents and read only the reply that the agent hands back. Current evaluation scores task completion and the faithfulness of reports, and it leaves open whether the requester can understand and use the reply. We call this property reply usability and measure it with a reader model that takes the requester's position and tests whether the reply can be interpreted from what the requester has seen, is in the requester's language and answers the request. Its agreement with the majority of five annotators lies within the range of the individual annotators. Across eight models from seven families, 54% of 480 replies fail, and the best model still fails on 30%. Failed replies mostly point to material the requester has not seen or use labels that the agent coined during its work. Stating explicit requirements for the reply reduces the failure rate from 0.52 to 0.43 together with errors and omissions, and a secondary model that rewrites the reply reduces it to 0.36, which human annotators confirm in a blind comparison. Reply usability can thus be scored alongside task success and improved with one appended message or one call to a secondary model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 30 of its 52 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

30 of 52 references.

References · 30

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  9. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  10. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  11. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  12. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
18 more
  1. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  2. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  3. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  4. 2021Decontextualization: Making Sentences Stand-Alonecited by 102
  5. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 65
  6. 2022Making Science Simple: Corpora for the Lay Summarisation of Scientific Literaturecited by 55
  7. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 44
  8. 2020Inquisitive Question Generation for High Level Text Comprehensioncited by 32
  9. 2025The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Modelscited by 29
  10. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 29
  11. 2023A Question Answering Framework for Decontextualizing User-facing Snippets from Scientific Documentscited by 16
  12. 2026How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessionscited by 14
  13. 2023Personalized Jargon Identification for Enhanced Interdisciplinary Communicationcited by 12
  14. 2026Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agentscited by 8
  15. 2026The Handoff Tax: Continuing Non-Native Trajectories in LLM Agentscited by 6
  16. 2026Handoff Debt: The Rediscovery Cost When Coding Agents Take Over Interrupted Taskscited by 3
  17. 2026Plans They Abandon, Reports They Author: The Narrative Layer of Autonomous Agentscited by 3
  18. 2026The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?cited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
  2. 2025Adaptive Multi-Agent Response Refinement in Conversational Systems
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
  4. 2026Feedback Indices to Evaluate LLM Responses to Rebuttals for Multiple Choice Type Questions
  5. 2026Do Agents Repair When Challenged -- or Just Reply? Challenge, Repair, and Public Correction in a Deployed Agent Forum
  6. 2026Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agents
  7. 2025Position: The Real Barrier to LLM Agent Usability is Agentic ROI
  8. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate