# Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models

arXiv: https://arxiv.org/abs/2208.07852 · HTML: https://mlsanity.com/paper/2208.07852/

## Key facts

- Cited by 70 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 16 Aug 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [GLTR: Statistical Detection and Visualization of Generated Text](https://mlsanity.com/paper/1906.04043/) (2019)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models](https://mlsanity.com/paper/2009.07053/) (2020)
- [Understanding Hidden Memories of Recurrent Neural Networks](https://mlsanity.com/paper/1710.10777/) (2017)
- [Dodrio: Exploring Transformer Models with Interactive Visualization](https://mlsanity.com/paper/2103.14625/) (2021)
- [VisQA: X-raying Vision and Language Reasoning in Transformers](https://mlsanity.com/paper/2104.00926/) (2021)
- [LMdiff: A Visual Diff Tool to Compare Language Models](https://mlsanity.com/paper/2111.01582/) (2021)
- [T3-Vis: a visual analytic framework for Training and fine-Tuning Transformers in NLP](https://mlsanity.com/paper/2108.13587/) (2021)

## Built on by (most cited first, 30 of 70)

- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [Prompting PaLM for Translation: Assessing Strategies and Performance](https://mlsanity.com/paper/2211.09102/) (2022)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost](https://mlsanity.com/paper/2407.19825/) (2024)
- [Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs](https://mlsanity.com/paper/2311.02262/) (2023)
- [ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing](https://mlsanity.com/paper/2309.09128/) (2023)
- [Getting pwn'd by AI: Penetration Testing with Large Language Models](https://mlsanity.com/paper/2308.00121/) (2023)
- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Effectively Controlling Reasoning Models through Thinking Intervention](https://mlsanity.com/paper/2503.24370/) (2025)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- ["HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media](https://mlsanity.com/paper/2304.10619/) (2023)
- [RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions](https://mlsanity.com/paper/2302.09466/) (2023)
- [Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thought](https://mlsanity.com/paper/2308.08614/) (2023)
- [Visual cognition in multimodal large language models](https://mlsanity.com/paper/2311.16093/) (2023)
- [Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting](https://mlsanity.com/paper/2303.03199/) (2023)
- [PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions](https://mlsanity.com/paper/2308.05184/) (2023)
- [ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles](https://mlsanity.com/paper/2310.15428/) (2023)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models](https://mlsanity.com/paper/2402.10524/) (2024)
- [The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiring](https://mlsanity.com/paper/2405.04412/) (2024)
- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [LLM4PLC: Harnessing Large Language Models for Verifiable Programming of PLCs in Industrial Control Systems](https://mlsanity.com/paper/2401.05443/) (2024)
- [Tree Prompting: Efficient Task Adaptation without Fine-Tuning](https://mlsanity.com/paper/2310.14034/) (2023)
- [Large Language Models as Software Components: A Taxonomy for LLM-Integrated Applications](https://mlsanity.com/paper/2406.10300/) (2024)
- [From Prompt Engineering to Prompt Science With Human in the Loop](https://mlsanity.com/paper/2401.04122/) (2024)
- [Exploring Prompt Engineering Practices in the Enterprise](https://mlsanity.com/paper/2403.08950/) (2024)
- [Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products](https://mlsanity.com/paper/2410.12071/) (2024)
- [Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges](https://mlsanity.com/paper/2308.00031/) (2023)

## Abstract (arXiv)

State-of-the-art neural language models can now be used to solve ad-hoc language tasks through zero-shot prompting without the need for supervised training. This approach has gained popularity in recent years, and researchers have demonstrated prompts that achieve strong accuracy on specific NLP tasks. However, finding a prompt for new tasks requires experimentation. Different prompt templates with different wording choices lead to significant accuracy differences. PromptIDE allows users to experiment with prompt variations, visualize prompt performance, and iteratively optimize prompts. We developed a workflow that allows users to first focus on model feedback using small data before moving on to a large data regime that allows empirical grounding of promising prompts using quantitative measures of the task. The tool then allows easy deployment of the newly created ad-hoc models. We demonstrate the utility of PromptIDE (demo at http://prompt.vizhub.ai) and our workflow using several real-world use cases.