When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Tasks
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Existing AI agents typically execute multi-step tasks autonomously and only allow user confirmation at the end. During execution, users have little control, making the confirm-at-end approach brittle: a single error can cascade and force a complete restart. Confirming every step avoids such failures, but imposes tedious overhead. Balancing excessive interruptions against costly rollbacks remains an open challenge. We address this problem by modeling confirmation as a minimum time scheduling problem. We conducted a formative study with eight participants, which revealed a recurring Confirmation-Diagnosis-Correction-Redo (CDCR) pattern in how users monitor errors. Based on this pattern, we developed a decision-theoretic model to determine time-efficient confirmation point placement. We then evaluated our approach using a within-subjects study where 48 participants monitored AI agents and repaired their mistakes while executing tasks. Results show that 81 percent of participants preferred our intermediate confirmation approach over the confirm-at-end approach used by existing systems, and task completion time was reduced by 13.54 percent.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 60 of its 123 references that are in the map. First posted on arXiv 6 Oct 2025.
Builds on
60 of 123 references.
Closest ancestors · 1
Same direction, newest first
Other references · 59
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
- 2022Survey of Hallucination in Natural Language Generationcited by 2,075
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2016Concrete Problems in AI Safetycited by 1,642
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
38 more
- 2021Carbon Emissions and Large Neural Network Trainingcited by 567
- 2024Understanding the planning of LLM agents: A surveycited by 421
- 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 398
- 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
- 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
- 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 354
- 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
- 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
- 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
- 2024AI Agents That Mattercited by 219
- 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
- 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 183
- 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
- 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 125
- 2024SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulationcited by 111
- 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 103
- 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
- 2024PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Taskscited by 82
- 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 65
- 2023Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creationcited by 64
- 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 63
- 2024Design Principles for Generative AI Applicationscited by 57
- 2023LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordinationcited by 48
- 2025Magentic-UI: Towards Human-in-the-loop Agentic Systemscited by 43
- 2024On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)cited by 40
- 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 39
- 2025CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigationcited by 36
- 2025Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Supportcited by 34
- 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
- 2021Assessing the Impact of Automated Suggestions on Decision Making: Domain Experts Mediate Model Errors but Take Less Initiativecited by 28
- 2024Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modelingcited by 25
- 2024Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertaintycited by 25
- 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 24
- 2022UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Librarycited by 20
- 2023Computers as Bad Social Actors: Dark Patterns and Anti-Patterns in Interfaces that Act Sociallycited by 20
- 2025Morae: Proactively Pausing UI Agents for User Choicescited by 19
- 2021Adapting User Interfaces with Model-based Reinforcement Learningcited by 17
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Hedwig: Dynamic Autonomy for Coding Agents Under Local Oversightcited by 3
- 2026A Human-AI Theorem Connecting Spontaneous and Field-Induced Mechanisms of Collective Behavior in One Dimensioncited by 0
- 2026Beyond Prompt-to-App: Accountable Translation in Teacher-Facing Agentic Authoringcited by 0
- 2026AI Soccer Analyst: Stage-Aware and Verifiable Human-AI Collaboration for Soccer Data Analysiscited by 0
- 2026Value-Sensitive Delegation in Everyday AI Agent Use: Evidence from OpenClawcited by 0
- 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0
- 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents30 Sep 2026
- 2026Value-Sensitive Delegation in Everyday AI Agent Use: Evidence from OpenClaw18 Sep 2026
- 2026AI Soccer Analyst: Stage-Aware and Verifiable Human-AI Collaboration for Soccer Data Analysis10 Sep 2026
- 2026Beyond Prompt-to-App: Accountable Translation in Teacher-Facing Agentic Authoring4 Sep 2026
- 2026A Human-AI Theorem Connecting Spontaneous and Field-Induced Mechanisms of Collective Behavior in One Dimension31 Aug 2026
- 2026Hedwig: Dynamic Autonomy for Coding Agents Under Local Oversight12 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026Overseeing Agents Without Constant Oversight: Challenges and Opportunities
- 2026Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops
- 2025On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows
- 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
- 2025E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
- 2026Confident and Wrong: Silent Semantic Failures in Coding Agents
- 2026Beyond Fluency: Toward Reliable Trajectories in Agentic IR
- 2026Beyond Component Testing: Validating Agentic AI Systems