# Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support

arXiv: https://arxiv.org/abs/2502.18658 · HTML: https://mlsanity.com/paper/2502.18658/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 81 references that are in the map.
- First posted on arXiv 25 Feb 2025.

## Builds on (most cited first, 18 of 18)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [Expanding Explainability: Towards Social Transparency in AI systems](https://mlsanity.com/paper/2101.04719/) (2021)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming](https://mlsanity.com/paper/2210.14306/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges](https://mlsanity.com/paper/2303.17125/) (2023)
- [How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment](https://mlsanity.com/paper/2309.14049/) (2023)
- [On the Design of AI-powered Code Assistants for Notebooks](https://mlsanity.com/paper/2301.11178/) (2023)
- [Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice](https://mlsanity.com/paper/2404.14901/) (2024)
- [Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programming](https://mlsanity.com/paper/2306.05153/) (2023)
- [Ivie: Lightweight Anchored Explanations of Just-Generated Code](https://mlsanity.com/paper/2403.02491/) (2024)
- [CoLadder: Supporting Programmers with Hierarchical Code Generation in Multi-Level Abstraction](https://mlsanity.com/paper/2310.08699/) (2023)
- [Understanding How Programmers Can Use Annotations on Documentation](https://mlsanity.com/paper/2111.08684/) (2021)

## Built on by (most cited first, 30 of 34)

- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems](https://mlsanity.com/paper/2604.14228/) (2026)
- [AIDev: Studying AI Coding Agents on GitHub](https://mlsanity.com/paper/2602.09185/) (2026)
- [Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight](https://mlsanity.com/paper/2509.10723/) (2025)
- [A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System](https://mlsanity.com/paper/2510.09721/) (2025)
- [ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices](https://mlsanity.com/paper/2507.21378/) (2025)
- [When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Tasks](https://mlsanity.com/paper/2510.05307/) (2025)
- [Interaction as Intelligence: Deep Research With Human-AI Partnership](https://mlsanity.com/paper/2507.15759/) (2025)
- [ClearFairy: Capturing Creative Workflows through Decision Structuring, In-Situ Questioning, and Rationale Inference](https://mlsanity.com/paper/2509.14537/) (2025)
- [GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks](https://mlsanity.com/paper/2603.25864/) (2026)
- [Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scale](https://mlsanity.com/paper/2504.09283/) (2025)
- [From Passive Tool to Socio-cognitive Teammate: A Conceptual Framework for Agentic AI in Human-AI Collaborative Learning](https://mlsanity.com/paper/2508.14825/) (2025)
- [Evalet: Evaluating Large Language Models through Functional Fragmentation](https://mlsanity.com/paper/2509.11206/) (2025)
- [Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes](https://mlsanity.com/paper/2510.04452/) (2025)
- [Optimizing LLM Code Suggestions: Feedback-Driven Timing with Lightweight State Bounds](https://mlsanity.com/paper/2511.18842/) (2025)
- [HiLDe: Intentional Code Generation via Human-in-the-Loop Decoding](https://mlsanity.com/paper/2505.22906/) (2025)
- [Human-Human-AI Triadic Programming: Uncovering the Role of AI Agent and the Value of Human Partner in Collaborative Learning](https://mlsanity.com/paper/2601.12134/) (2026)
- ["When to Hand Off, When to Work Together": Understanding Concurrent Human-Agent Interaction in Shared Co-Creative Workspaces](https://mlsanity.com/paper/2603.02050/) (2026)
- [Beyond the Desk: Barriers and Future Opportunities for AI to Assist Scientists in Embodied Physical Tasks](https://mlsanity.com/paper/2603.19504/) (2026)
- [Comparing Developer and LLM Biases in Code Evaluation](https://mlsanity.com/paper/2603.24586/) (2026)
- [CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks](https://mlsanity.com/paper/2606.09833/) (2026)
- [Designing Proactive Thought Partners for Writing](https://mlsanity.com/paper/2609.01588/) (2026)
- [Choose Your Own Adventure: Non-Linear AI-Assisted Programming with EvoGraph](https://mlsanity.com/paper/2604.18883/) (2026)
- [Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration](https://mlsanity.com/paper/2606.06388/) (2026)
- [Human-AI Coordination Zones: A Framework for Designing Human-in-the-Loop Experiences with Agentic AI](https://mlsanity.com/paper/2606.09848/) (2026)
- [LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs](https://mlsanity.com/paper/2608.06041/) (2026)
- [VeriForge: Mitigating Latent Knowledge Gaps in Narrative Drafting via Mixed-Initiative Scaffolding](https://mlsanity.com/paper/2608.09698/) (2026)
- [Generation of Web Apps with Agentic IDEs: An Empirical Assessment](https://mlsanity.com/paper/2608.20903/) (2026)
- [VibeJam: A User Study Platform for Web Development with Agents](https://mlsanity.com/paper/2608.29889/) (2026)
- [Towards Cognitive Process-Aware Proactive Writing Support](https://mlsanity.com/paper/2608.30424/) (2026)

## Abstract (arXiv)

AI programming tools enable powerful code generation, and recent prototypes attempt to reduce user effort with proactive AI agents, but their impact on programming workflows remains unexplored. We introduce and evaluate Codellaborator, a design probe LLM agent that initiates programming assistance based on editor activities and task context. We explored three interface variants to assess trade-offs between increasingly salient AI support: prompt-only, proactive agent, and proactive agent with presence and context (Codellaborator). In a within-subject study (N=18), we find that proactive agents increase efficiency compared to prompt-only paradigm, but also incur workflow disruptions. However, presence indicators and interaction context support alleviated disruptions and improved users' awareness of AI processes. We underscore trade-offs of Codellaborator on user control, ownership, and code understanding, emphasizing the need to adapt proactivity to programming processes. Our research contributes to the design exploration and evaluation of proactive AI systems, presenting design implications on AI-integrated programming workflow.