# Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfaces

arXiv: https://arxiv.org/abs/2309.14459 · HTML: https://mlsanity.com/paper/2309.14459/

## Key facts

- Cited by 72 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2023.

## Built on by (most cited first, 30 of 72)

- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization](https://mlsanity.com/paper/2408.01703/) (2024)
- [Patchview: LLM-Powered Worldbuilding with Generative Dust and Magnet Visualization](https://mlsanity.com/paper/2408.04112/) (2024)
- [VeriPlan: Integrating Formal Verification and LLMs into End-User Planning](https://mlsanity.com/paper/2502.17898/) (2025)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses](https://mlsanity.com/paper/2406.04755/) (2024)
- [Thinking beyond the anthropomorphic paradigm benefits LLM research](https://mlsanity.com/paper/2502.09192/) (2025)
- [Measuring and mitigating overreliance to build human-compatible AI](https://mlsanity.com/paper/2509.08010/) (2025)
- [HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants](https://mlsanity.com/paper/2509.08494/) (2025)
- [Human-AI Co-design for Clinical Prediction Models](https://mlsanity.com/paper/2601.09072/) (2026)
- [AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools](https://mlsanity.com/paper/2502.18736/) (2025)
- [CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science Experiments](https://mlsanity.com/paper/2509.13588/) (2025)
- [Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency](https://mlsanity.com/paper/2402.19366/) (2024)
- [OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models](https://mlsanity.com/paper/2508.21061/) (2025)
- [Just-In-Time Objectives: A General Approach for Specialized AI Interactions](https://mlsanity.com/paper/2510.14591/) (2025)
- [AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild](https://mlsanity.com/paper/2602.11750/) (2026)
- [DesignWeaver: Dimensional Scaffolding for Text-to-Image Product Design](https://mlsanity.com/paper/2502.09867/) (2025)
- [Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows](https://mlsanity.com/paper/2502.18737/) (2025)
- [Exploring the Potential of Metacognitive Support Agents for Human-AI Co-Creation](https://mlsanity.com/paper/2506.12879/) (2025)
- [DiscoverLLM: From Executing Intents to Discovering Them](https://mlsanity.com/paper/2602.03429/) (2026)
- [Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software](https://mlsanity.com/paper/2504.15549/) (2025)
- [Orca: Browsing at Scale Through User-Driven and AI-Facilitated Orchestration Across Malleable Webpages](https://mlsanity.com/paper/2505.22831/) (2025)
- [From Words to Widgets for Controllable LLM Generation](https://mlsanity.com/paper/2604.10925/) (2026)
- [AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use](https://mlsanity.com/paper/2607.20536/) (2026)
- [CogErgLLM: Exploring Large Language Model Systems Design Perspective Using Cognitive Ergonomics](https://mlsanity.com/paper/2407.02885/) (2024)
- [From Consumption to Collaboration: Measuring Interaction Patterns to Augment Human Cognition in Open-Ended Tasks](https://mlsanity.com/paper/2504.02780/) (2025)
- [Expanding the Generative AI Design Space through Structured Prompting and Multimodal Interfaces](https://mlsanity.com/paper/2504.14320/) (2025)
- [NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification](https://mlsanity.com/paper/2508.02823/) (2025)
- [Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agents](https://mlsanity.com/paper/2509.12049/) (2025)
- [ALLOY: Generating Reusable Agent Workflows from User Demonstration](https://mlsanity.com/paper/2510.10049/) (2025)

## Abstract (arXiv)

Large language models (LLMs) exhibit dynamic capabilities and appear to comprehend complex and ambiguous natural language prompts. However, calibrating LLM interactions is challenging for interface designers and end-users alike. A central issue is our limited grasp of how human cognitive processes begin with a goal and form intentions for executing actions, a blindspot even in established interaction models such as Norman's gulfs of execution and evaluation. To address this gap, we theorize how end-users 'envision' translating their goals into clear intentions and craft prompts to obtain the desired LLM response. We define a process of Envisioning by highlighting three misalignments: (1) knowing whether LLMs can accomplish the task, (2) how to instruct the LLM to do the task, and (3) how to evaluate the success of the LLM's output in meeting the goal. Finally, we make recommendations to narrow the envisioning gulf in human-LLM interactions.