# ScatterShot: Interactive In-context Example Curation for Text Transformation

arXiv: https://arxiv.org/abs/2302.07346 · HTML: https://mlsanity.com/paper/2302.07346/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 73 references that are in the map.
- First posted on arXiv 14 Feb 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Selective Annotation Makes Language Models Better Few-Shot Learners](https://mlsanity.com/paper/2209.01975/) (2022)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Want To Reduce Labeling Cost? GPT-3 Can Help](https://mlsanity.com/paper/2108.13487/) (2021)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [In-context Examples Selection for Machine Translation](https://mlsanity.com/paper/2212.02437/) (2022)
- [Domino: Discovering Systematic Errors with Cross-Modal Embeddings](https://mlsanity.com/paper/2203.14960/) (2022)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension](https://mlsanity.com/paper/2002.00293/) (2020)
- [Batch Active Learning at Scale](https://mlsanity.com/paper/2107.14263/) (2021)
- [Deep Active Learning over the Long Tail](https://mlsanity.com/paper/1711.00941/) (2017)

## Built on by (most cited first, 20 of 20)

- [EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria](https://mlsanity.com/paper/2309.13633/) (2023)
- [Prompt2Model: Generating Deployable Models from Natural Language Instructions](https://mlsanity.com/paper/2308.12261/) (2023)
- [ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles](https://mlsanity.com/paper/2310.15428/) (2023)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Students](https://mlsanity.com/paper/2410.04078/) (2024)
- [Does Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts?](https://mlsanity.com/paper/2304.01002/) (2023)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products](https://mlsanity.com/paper/2410.12071/) (2024)
- [On-the-fly Definition Augmentation of LLMs for Biomedical NER](https://mlsanity.com/paper/2404.00152/) (2024)
- [PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers' Workflows](https://mlsanity.com/paper/2310.15435/) (2023)
- [OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models](https://mlsanity.com/paper/2508.21061/) (2025)
- [Evalet: Evaluating Large Language Models through Functional Fragmentation](https://mlsanity.com/paper/2509.11206/) (2025)
- [Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning](https://mlsanity.com/paper/2408.03819/) (2024)
- [Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation](https://mlsanity.com/paper/2507.02306/) (2025)
- [POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models](https://mlsanity.com/paper/2406.03843/) (2024)
- [Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behavior](https://mlsanity.com/paper/2510.12728/) (2025)
- [A Visual Tool for Interactive Model Explanation using Sensitivity Analysis](https://mlsanity.com/paper/2508.04269/) (2025)
- [Human-AI Interaction Alignment: Designing, Evaluating, and Evolving Value-Centered AI For Reciprocal Human-AI Futures](https://mlsanity.com/paper/2512.21551/) (2025)
- [ExPerT: Personalizing LLM Responses to Users' Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues](https://mlsanity.com/paper/2607.01242/) (2026)

## Abstract (arXiv)

The in-context learning capabilities of LLMs like GPT-3 allow annotators to customize an LLM to their specific tasks with a small number of examples. However, users tend to include only the most obvious patterns when crafting examples, resulting in underspecified in-context functions that fall short on unseen cases. Further, it is hard to know when "enough" examples have been included even for known patterns. In this work, we present ScatterShot, an interactive system for building high-quality demonstration sets for in-context learning. ScatterShot iteratively slices unlabeled data into task-specific patterns, samples informative inputs from underexplored or not-yet-saturated slices in an active learning manner, and helps users label more efficiently with the help of an LLM and the current example set. In simulation studies on two text perturbation scenarios, ScatterShot sampling improves the resulting few-shot functions by 4-5 percentage points over random sampling, with less variance as more examples are added. In a user study, ScatterShot greatly helps users in covering different patterns in the input space and labeling in-context examples more efficiently, resulting in better in-context learning and less user effort.