# Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordances

arXiv: https://arxiv.org/abs/2307.01142 · HTML: https://mlsanity.com/paper/2307.01142/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jul 2023.

## Built on by (most cited first, 7 of 7)

- [Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents](https://mlsanity.com/paper/2308.01542/) (2023)
- [From Words to Widgets for Controllable LLM Generation](https://mlsanity.com/paper/2604.10925/) (2026)
- [Towards Full Authorship with AI: Supporting Revision with AI-Generated Views](https://mlsanity.com/paper/2403.01055/) (2024)
- [Survey of User Interface Design and Interaction Techniques in Generative AI Applications](https://mlsanity.com/paper/2410.22370/) (2024)
- [Enhancing Programming Error Messages in Real Time with Generative AI](https://mlsanity.com/paper/2402.08072/) (2024)
- [CoGen: Creation of Reusable UI Components in Figma via Textual Commands](https://mlsanity.com/paper/2601.10536/) (2026)
- [Alignment has a Fantasia Problem](https://mlsanity.com/paper/2604.21827/) (2026)

## Abstract (arXiv)

To help users do complex work, researchers have developed techniques to integrate AI and human intelligence into user interfaces (UIs). With the recent introduction of large language models (LLMs), which can generate text in response to a natural language prompt, there are new opportunities to consider how to integrate LLMs into UIs. We present Prompt Middleware, a framework for generating prompts for LLMs based on UI affordances. These include prompts that are predefined by experts (static prompts), generated from templates with fill-in options in the UI (template-based prompts), or created from scratch (free-form prompts). We demonstrate this framework with FeedbackBuffet, a writing assistant that automatically generates feedback based on a user's text input. Inspired by prior research showing how templates can help non-experts perform more like experts, FeedbackBuffet leverages template-based prompt middleware to enable feedback seekers to specify the types of feedback they want to receive as options in a UI. These options are composed using a template to form a feedback request prompt to GPT-3. We conclude with a discussion about how Prompt Middleware can help developers integrate LLMs into UIs.