# Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software

arXiv: https://arxiv.org/abs/2504.15549 · HTML: https://mlsanity.com/paper/2504.15549/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 90 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces](https://mlsanity.com/paper/2309.14459/) (2023)
- ["Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction](https://mlsanity.com/paper/2210.03735/) (2022)
- [Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works](https://mlsanity.com/paper/2210.08477/) (2022)
- [In-IDE Code Generation from Natural Language: Promise and Challenges](https://mlsanity.com/paper/2101.11149/) (2021)
- [What is it like to program with artificial intelligence?](https://mlsanity.com/paper/2208.06213/) (2022)
- [Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition](https://mlsanity.com/paper/2407.02651/) (2024)
- [Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop](https://mlsanity.com/paper/1703.08428/) (2017)
- [Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking](https://mlsanity.com/paper/2402.08030/) (2024)
- [The Rise of the AI Co-Pilot: Lessons for Design from Aviation and Beyond](https://mlsanity.com/paper/2311.14713/) (2023)
- [How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot](https://mlsanity.com/paper/2208.14613/) (2022)
- [Ivie: Lightweight Anchored Explanations of Just-Generated Code](https://mlsanity.com/paper/2403.02491/) (2024)

## Built on by (most cited first, 5 of 5)

- [GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks](https://mlsanity.com/paper/2603.25864/) (2026)
- [Plover: Steering GUI Agents through Plan-Centric Interaction](https://mlsanity.com/paper/2607.15193/) (2026)
- ["When to Hand Off, When to Work Together": Understanding Concurrent Human-Agent Interaction in Shared Co-Creative Workspaces](https://mlsanity.com/paper/2603.02050/) (2026)
- [Helping the Helper: LLM-Assisted Problem Articulation for Older Adults Seeking Technology Support](https://mlsanity.com/paper/2601.10018/) (2026)
- [Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation](https://mlsanity.com/paper/2604.14668/) (2026)

## Abstract (arXiv)

Large Language Model (LLM)-based in-application assistants, or copilots, can automate software tasks, but users often prefer learning by doing, raising questions about the optimal level of automation for an effective user experience. We investigated two automation paradigms by designing and implementing a fully automated copilot (AutoCopilot) and a semi-automated copilot (GuidedCopilot) that automates trivial steps while offering step-by-step visual guidance. In a user study (N=20) across data analysis and visual design tasks, GuidedCopilot outperformed AutoCopilot in user control, software utility, and learnability, especially for exploratory and creative tasks, while AutoCopilot saved time for simpler visual tasks. A follow-up design exploration (N=10) enhanced GuidedCopilot with task-and state-aware features, including in-context preview clips and adaptive instructions. Our findings highlight the critical role of user control and tailored guidance in designing the next generation of copilots that enhance productivity, support diverse skill levels, and foster deeper software engagement.