mlsanityData to 14 Oct 2026

DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coaching

arXiv 2606.31980 ↗First posted 30 Jun 2026cs.CL cs.AI cs.HCcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Agents are increasingly capable of automating software tasks, but can they teach humans how to use software themselves? We introduce DigitalCoach, a multimodal dataset of 72 human expert-novice computer use coaching sessions consisting of 22,752 dialogue turns grounded in 28.1 hours of screen and input event recordings across five software applications. We use DigitalCoach to evaluate whether state-of-the-art models can teach humans how to use computers. Automated evaluation shows that models differ from humans in how they coach: models provide more direct instructions, but fewer explanations, error diagnoses, and knowledge-check questions. When we fix the coaching method, models produce utterances similar to human references yet poorly grounded in visual context. Interactive evaluation confirms that model coaches cause learners to passively follow instructions without deeper engagement and fall short in visual grounding. DigitalCoach lays a foundation for collaborative and proactive computer use coaching agents. Data and code are available at https://project-digital-coach.vercel.app.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 29 of its 55 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

29 of 55 references.

References · 29

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  3. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  4. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  5. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  6. 2024GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editingcited by 133
  7. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 131
  8. 2019Executing Instructions in Situated Collaborative Interactionscited by 69
  9. 2021MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Taskscited by 62
  10. 2019Jointly Measuring Diversity and Quality in Text Generation Modelscited by 61
  11. 2025MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutorscited by 49
  12. 2026Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoningcited by 49
17 more
  1. 2024VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Taskscited by 43
  2. 2025CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigationcited by 36
  3. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  4. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 28
  5. 2025RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Userscited by 21
  6. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 19
  7. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 19
  8. 2024Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMscited by 18
  9. 2019Quality Estimation for Image Captions Based on Large-scale Human Evaluationscited by 17
  10. 2025Vid2Coach: Transforming How-To Videos into Task Assistantscited by 12
  11. 2019An Annotated Corpus of Reference Resolution for Interpreting Common Groundingcited by 6
  12. 2022What's in a Caption? Dataset-Specific Linguistic Diversity and Its Effect on Visual Description Models and Metricscited by 6
  13. 2025Completion neq Collaboration: Scaling Collaborative Effort with Agentscited by 6
  14. 2026GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Taskscited by 4
  15. 2025Characterizing Language Use in a Collaborative Situated Gamecited by 3
  16. 2025Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Ozcited by 2
  17. 2025Generating Dialogues from Egocentric Instructional Videos for Task Assistance: Dataset, Method and Benchmarkcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Demo2Tutorial: From Human Experience to Multimodal Software Tutorials
  2. 2026Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction
  3. 2025Grounding Computer Use Agents on Human Demonstrations
  4. 2024Coaching Copilot: Blended Form of an LLM-Powered Chatbot and a Human Coach to Effectively Support Self-Reflection for Leadership Growth
  5. 2026Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration
  6. 2025Efficient Agent Training for Computer Use
  7. 2026HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
  8. 2026AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks