# DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models

arXiv: https://arxiv.org/abs/2310.03691 · HTML: https://mlsanity.com/paper/2310.03691/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2023.

## Built on by (most cited first, 30 of 32)

- [WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization](https://mlsanity.com/paper/2408.01703/) (2024)
- [Data Formulator 2: Iterative Creation of Data Visualizations, with AI Transforming Data Along the Way](https://mlsanity.com/paper/2408.16119/) (2024)
- [LightVA: Lightweight Visual Analytics with LLM Agent-Based Task Planning and Execution](https://mlsanity.com/paper/2411.05651/) (2024)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions](https://mlsanity.com/paper/2503.04110/) (2025)
- [PosterMate: Audience-driven Collaborative Persona Agents for Poster Design](https://mlsanity.com/paper/2507.18572/) (2025)
- [AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools](https://mlsanity.com/paper/2502.18736/) (2025)
- [Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends](https://mlsanity.com/paper/2606.01164/) (2026)
- [OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models](https://mlsanity.com/paper/2508.21061/) (2025)
- [Data Analysis in the Era of Generative AI](https://mlsanity.com/paper/2409.18475/) (2024)
- [Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows](https://mlsanity.com/paper/2502.18737/) (2025)
- [Exploring Multimodal Prompt for Visualization Authoring with Large Language Models](https://mlsanity.com/paper/2504.13700/) (2025)
- [Collaborative Document Editing with Multiple Users and AI Agents](https://mlsanity.com/paper/2509.11826/) (2025)
- [Survey of User Interface Design and Interaction Techniques in Generative AI Applications](https://mlsanity.com/paper/2410.22370/) (2024)
- [The Value-Sensitive Conversational Agent Co-Design Framework](https://mlsanity.com/paper/2310.11848/) (2023)
- [Composable Prompting Workspaces for Creative Writing: Exploration and Iteration Using Dynamic Widgets](https://mlsanity.com/paper/2503.21394/) (2025)
- [MapStory: Prototyping Editable Map Animations with LLM Agents](https://mlsanity.com/paper/2505.21966/) (2025)
- [NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification](https://mlsanity.com/paper/2508.02823/) (2025)
- [StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagation](https://mlsanity.com/paper/2508.03182/) (2025)
- [Texterial: A Text-as-Material Interaction Paradigm for LLM-Mediated Writing](https://mlsanity.com/paper/2603.00452/) (2026)
- [How to Steer Your Multi-Agent System: Human-LLM Collaborative Planning](https://mlsanity.com/paper/2605.23023/) (2026)
- [ClickDiffusion: Harnessing LLMs for Interactive Precise Image Editing](https://mlsanity.com/paper/2404.04376/) (2024)
- [Exploring Mobile Touch Interaction with Large Language Models](https://mlsanity.com/paper/2502.07629/) (2025)
- [ChatGPT and U(X): A Rapid Review on Measuring the User Experience](https://mlsanity.com/paper/2503.15808/) (2025)
- [Bridging Gulfs in UI Generation through Semantic Guidance](https://mlsanity.com/paper/2601.19171/) (2026)
- [The Keyhole Effect: Why Chat Interfaces Fail at Data Analysis](https://mlsanity.com/paper/2602.00947/) (2026)
- [PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR](https://mlsanity.com/paper/2603.02366/) (2026)
- [TrICy: Trigger-guided Data-to-text Generation with Intent aware Attention-Copy](https://mlsanity.com/paper/2402.01714/) (2024)
- [VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents](https://mlsanity.com/paper/2506.21582/) (2025)
- [Insights into User Interface Innovations from a Design Thinking Workshop at deRSE25](https://mlsanity.com/paper/2508.18784/) (2025)

## Abstract (arXiv)

We characterize and demonstrate how the principles of direct manipulation can improve interaction with large language models. This includes: continuous representation of generated objects of interest; reuse of prompt syntax in a toolbar of commands; manipulable outputs to compose or control the effect of prompts; and undo mechanisms. This idea is exemplified in DirectGPT, a user interface layer on top of ChatGPT that works by transforming direct manipulation actions to engineered prompts. A study shows participants were 50% faster and relied on 50% fewer and 72% shorter prompts to edit text, code, and vector images compared to baseline ChatGPT. Our work contributes a validated approach to integrate LLMs into traditional software using direct manipulation. Data, code, and demo available at https://osf.io/3wt6s.