# IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models

arXiv: https://arxiv.org/abs/2310.10873 · HTML: https://mlsanity.com/paper/2310.10873/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 97 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 30 of 61)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Domain Generalization: A Survey](https://mlsanity.com/paper/2103.02503/) (2021)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Dataset Condensation with Gradient Matching](https://mlsanity.com/paper/2006.05929/) (2020)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Trained Transformers Learn Linear Models In-Context](https://mlsanity.com/paper/2306.09927/) (2023)

## Built on by (most cited first, 27 of 27)

- [MotionLLM: Understanding Human Behaviors from Human Motions and Videos](https://mlsanity.com/paper/2405.20340/) (2024)
- [StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows](https://mlsanity.com/paper/2403.11322/) (2024)
- [One-Shot Learning as Instruction Data Prospector for Large Language Models](https://mlsanity.com/paper/2312.10302/) (2023)
- [Adaptive In-conversation Team Building for Language Model Agents](https://mlsanity.com/paper/2405.19425/) (2024)
- [DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception](https://mlsanity.com/paper/2405.15232/) (2024)
- [Measuring and Reducing LLM Hallucination without Gold-Standard Answers](https://mlsanity.com/paper/2402.10412/) (2024)
- [RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios](https://mlsanity.com/paper/2412.08972/) (2024)
- [LaVin-DiT: Large Vision Diffusion Transformer](https://mlsanity.com/paper/2411.11505/) (2024)
- [Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling](https://mlsanity.com/paper/2410.03735/) (2024)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interface](https://mlsanity.com/paper/2410.00079/) (2024)
- [Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler](https://mlsanity.com/paper/2510.27172/) (2025)
- [ERASE: Error-Resilient Representation Learning on Graphs for Label Noise Tolerance](https://mlsanity.com/paper/2312.08852/) (2023)
- [OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis](https://mlsanity.com/paper/2501.04561/) (2025)
- [Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning](https://mlsanity.com/paper/2406.11890/) (2024)
- [Enhancing Cross-task Transfer of Large Language Models via Activation Steering](https://mlsanity.com/paper/2507.13236/) (2025)
- [DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning](https://mlsanity.com/paper/2405.14899/) (2024)
- [Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation](https://mlsanity.com/paper/2407.05693/) (2024)
- [Efficient and Effective Model Extraction](https://mlsanity.com/paper/2409.14122/) (2024)
- [Memory-Augmented Agent Training for Business Document Understanding](https://mlsanity.com/paper/2412.15274/) (2024)
- [General Phrase Debiaser: Debiasing Masked Language Models at a Multi-Token Level](https://mlsanity.com/paper/2311.13892/) (2023)
- [Evolutionary Pre-Prompt Optimization for Mathematical Reasoning](https://mlsanity.com/paper/2412.04291/) (2024)
- [VIOLA: Towards Video In-Context Learning with Minimal Annotations](https://mlsanity.com/paper/2601.15549/) (2026)
- [Off-the-shelf ChatGPT is a Good Few-shot Human Motion Predictor](https://mlsanity.com/paper/2405.15267/) (2024)
- [InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity](https://mlsanity.com/paper/2508.21003/) (2025)
- [SAIR: Cost-Efficient Multi-Stage ML Pipeline Autoscaling via In-Context Reinforcement Learning](https://mlsanity.com/paper/2601.22397/) (2026)
- [Verb-ICL: Rethinking In-Context Learning for Structured Prediction](https://mlsanity.com/paper/2610.04725/) (2026)

## Abstract (arXiv)

In-context learning is a promising paradigm that utilizes in-context examples as prompts for the predictions of large language models. These prompts are crucial for achieving strong performance. However, since the prompts need to be sampled from a large volume of annotated examples, finding the right prompt may result in high annotation costs. To address this challenge, this paper introduces an influence-driven selective annotation method that aims to minimize annotation costs while improving the quality of in-context examples. The essence of our method is to select a pivotal subset from a large-scale unlabeled data pool to annotate for the subsequent sampling of prompts. Specifically, a directed graph is first constructed to represent unlabeled data. Afterward, the influence of candidate unlabeled subsets is quantified with a diffusion process. A simple yet effective greedy algorithm for unlabeled data selection is lastly introduced. It iteratively selects the data if it provides a maximum marginal gain with respect to quantified influence. Compared with previous efforts on selective annotations, our influence-driven method works in an end-to-end manner, avoids an intractable explicit balance between data diversity and representativeness, and enjoys theoretical support. Experiments confirm the superiority of the proposed method on various benchmarks, achieving better performance under lower time consumption during subset selection. The project page is available at https://skzhang1.github.io/IDEAL/.