mlsanityData to 7 Oct 2026

DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models

arXiv 2310.03691 ↗First posted 5 Oct 2023cs.HCcited by 32

Not on the map. Why?

Abstract (arXiv)

We characterize and demonstrate how the principles of direct manipulation can improve interaction with large language models. This includes: continuous representation of generated objects of interest; reuse of prompt syntax in a toolbar of commands; manipulable outputs to compose or control the effect of prompts; and undo mechanisms. This idea is exemplified in DirectGPT, a user interface layer on top of ChatGPT that works by transforming direct manipulation actions to engineered prompts. A study shows participants were 50% faster and relied on 50% fewer and 72% shorter prompts to edit text, code, and vector images compared to baseline ChatGPT. Our work contributes a validated approach to integrate LLMs into traditional software using direct manipulation. Data, code, and demo available at https://osf.io/3wt6s.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Oct 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 16; 2026: 8.

32 papers.

Sort:

By area:

  1. 2024WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualizationcited by 38
  2. 2024LightVA: Lightweight Visual Analytics with LLM Agent-Based Task Planning and Executioncited by 20
  3. 2024Data Formulator 2: Iterative Creation of Data Visualizations, with AI Transforming Data Along the Waycited by 19
  4. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  5. 2025InterChat: Enhancing Generative Visual Analytics using Multimodal Interactionscited by 13
  6. 2025PosterMate: Audience-driven Collaborative Persona Agents for Poster Designcited by 9
  7. 2025AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Toolscited by 8
  8. 2025OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Modelscited by 7
  9. 2024Data Analysis in the Era of Generative AIcited by 6
  10. 2025Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflowscited by 6
  11. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 6
  12. 2025Exploring Multimodal Prompt for Visualization Authoring with Large Language Modelscited by 5
20 more
  1. 2025Collaborative Document Editing with Multiple Users and AI Agentscited by 5
  2. 2023The Value-Sensitive Conversational Agent Co-Design Frameworkcited by 3
  3. 2024Survey of User Interface Design and Interaction Techniques in Generative AI Applicationscited by 3
  4. 2025Composable Prompting Workspaces for Creative Writing: Exploration and Iteration Using Dynamic Widgetscited by 3
  5. 2025MapStory: Prototyping Editable Map Animations with LLM Agentscited by 3
  6. 2025NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modificationcited by 3
  7. 2025StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagationcited by 2
  8. 2026Texterial: A Text-as-Material Interaction Paradigm for LLM-Mediated Writingcited by 2
  9. 2026How to Steer Your Multi-Agent System: Human-LLM Collaborative Planningcited by 2
  10. 2024ClickDiffusion: Harnessing LLMs for Interactive Precise Image Editingcited by 1
  11. 2025Exploring Mobile Touch Interaction with Large Language Modelscited by 1
  12. 2025ChatGPT and U(X): A Rapid Review on Measuring the User Experiencecited by 1
  13. 2026Bridging Gulfs in UI Generation through Semantic Guidancecited by 1
  14. 2026The Keyhole Effect: Why Chat Interfaces Fail at Data Analysiscited by 1
  15. 2026PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XRcited by 1
  16. 2024TrICy: Trigger-guided Data-to-text Generation with Intent aware Attention-Copycited by 0
  17. 2025VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agentscited by 0
  18. 2025Insights into User Interface Innovations from a Design Thinking Workshop at deRSE25cited by 0
  19. 2026AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysiscited by 0
  20. 2026Point, Revise, Review: Grounded Agentic Analysis in Reactive Notebooks with marimo-lenscited by 0