Survey of User Interface Design and Interaction Techniques in Generative AI Applications
Not on the map. Why?
Abstract (arXiv)
The applications of generative AI have become extremely impressive, and the interplay between users and AI is even more so. Current human-AI interaction literature has taken a broad look at how humans interact with generative AI, but it lacks specificity regarding the user interface designs and patterns used to create these applications. Therefore, we present a survey that comprehensively presents taxonomies of how a human interacts with AI and the user interaction patterns designed to meet the needs of a variety of relevant use cases. We focus primarily on user-guided interactions, surveying interactions that are initiated by the user and do not include any implicit signals given by the user. With this survey, we aim to create a compendium of different user-interaction patterns that can be used as a reference for designers and developers alike. In doing so, we also strive to lower the entry barrier for those attempting to learn more about the design of generative AI applications.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 96 of its 175 references that are in the map. First posted on arXiv 28 Oct 2024.
Builds on
96 of 175 references.
References · 96
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023GPT-4 Technical Reportcited by 18,132
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2022Mass-Editing Memory in a Transformercited by 976
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
- 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
38 more
- 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
- 2023In-Context Retrieval-Augmented Language Modelscited by 694
- 2023Bias and Fairness in Large Language Models: A Surveycited by 626
- 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
- 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
- 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
- 2023MusicLM: Generating Music From Textcited by 442
- 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
- 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
- 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
- 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
- 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
- 2022Optimizing Prompts for Text-to-Image Generationcited by 204
- 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
- 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 197
- 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
- 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
- 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 145
- 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
- 2017ParlAI: A Dialog Research Software Platformcited by 139
- 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
- 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 117
- 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
- 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
- 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 95
- 2016DeepBach: a Steerable Model for Bach Chorales Generationcited by 90
- 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 90
- 2021Sparks: Inspiration for Science Writing using Language Modelscited by 80
- 2023A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challengescited by 75
- 2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 65
- 2023ChatBridge: Bridging Modalities with Large Language Model as a Language Catalystcited by 64
- 2023"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmerscited by 62
- 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 61
- 2022Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systemscited by 59
- 2022ClipFace: Text-guided Editing of Textured 3D Morphable Modelscited by 59
- 2023RealFill: Reference-Driven Generation for Authentic Image Completioncited by 58
- 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
- 2024Design Principles for Generative AI Applicationscited by 57
50 most cited of 96 All 96 (JSON)
Built on by
Citing papers per year: 2025: 3.
3 papers.