mlsanityData to 14 Oct 2026

AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools

arXiv 2502.18736 ↗First posted 26 Feb 2025cs.HC cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Chat-based prompts respond with verbose linear-sequential texts, making it difficult to explore and refine ambiguous intents, back up and reinterpret, or shift directions in creative AI-assisted design work. AI-Instruments instead embody "prompts" as interface objects via three key principles: (1) Reification of user-intent as reusable direct-manipulation instruments; (2) Reflection of multiple interpretations of ambiguous user-intents (Reflection-in-intent) as well as the range of AI-model responses (Reflection-in-response) to inform design "moves" towards a desired result; and (3) Grounding to instantiate an instrument from an example, result, or extrapolation directly from another instrument. Further, AI-Instruments leverage LLM's to suggest, vary, and refine new instruments, enabling a system that goes beyond hard-coded functionality by generating its own instrumental controls from content. We demonstrate four technology probes, applied to image generation, and qualitative insights from twelve participants, showing how AI-Instruments address challenges of intent formulation, steering via direct manipulation, and non-linear iterative workflows to reflect and resolve ambiguous intents.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 16 of its 77 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

16 of 77 references.

References · 16

  1. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
  4. 2021A Survey of Human-in-the-loop for Machine Learningcited by 215
  5. 2023Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfacescited by 72
  6. 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
  7. 2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 66
  8. 2023Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creationcited by 64
  9. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 62
  10. 2023Concept Decomposition for Visual Exploration and Inspirationcited by 61
  11. 2023Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Modelscited by 48
  12. 2023PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactionscited by 34
4 more
  1. 2023CreativeConnect: Supporting Reference Recombination for Graphic Design Ideation with Generative AIcited by 34
  2. 2023DirectGPT: A Direct Manipulation Interface to Interact with Large Language Modelscited by 32
  3. 2024Supporting Sensemaking of Large Language Model Outputs at Scalecited by 27
  4. 2023Prompting for Discovery: Flexible Sense-Making for AI Art-Making with Dreamsheetscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026DiscoverLLM: From Executing Intents to Discovering Themcited by 6
  2. 2025ClearFairy: Capturing Creative Workflows through Decision Structuring, In-Situ Questioning, and Rationale Inferencecited by 4
  3. 2025MagicQuillV2: Precise and Interactive Image Editing with Layered Visual Cuescited by 4
  4. 2026Texterial: A Text-as-Material Interaction Paradigm for LLM-Mediated Writingcited by 2
  5. 2026Bridging Gulfs in UI Generation through Semantic Guidancecited by 1
  6. 2026PersonaMail: Learning and Adapting Personal Communication Preferences for Context-Aware Email Writingcited by 1
  7. 2026PromptHelper: A Prompt Recommender System for Encouraging Creativity in AI Chatbot Interactionscited by 0
  8. 2026CogInstrument: Modeling Cognitive Processes for Bidirectional Human-LLM Alignment in Planning Taskscited by 0