# One-Shot Learning as Instruction Data Prospector for Large Language Models

arXiv: https://arxiv.org/abs/2312.10302 · HTML: https://mlsanity.com/paper/2312.10302/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 67 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 30 of 48)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)

## Built on by (most cited first, 30 of 43)

- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Rho-1: Not All Tokens Are What You Need](https://mlsanity.com/paper/2404.07965/) (2024)
- [DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving](https://mlsanity.com/paper/2407.13690/) (2024)
- [A Survey on Data Selection for LLM Instruction Tuning](https://mlsanity.com/paper/2402.05123/) (2024)
- [What is Wrong with Perplexity for Long-context Language Modeling?](https://mlsanity.com/paper/2410.23771/) (2024)
- [UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility](https://mlsanity.com/paper/2501.02341/) (2025)
- [Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection](https://mlsanity.com/paper/2402.12501/) (2024)
- [JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models](https://mlsanity.com/paper/2405.14365/) (2024)
- [SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection](https://mlsanity.com/paper/2402.16705/) (2024)
- [SAE-V: Interpreting Multimodal Models for Enhanced Alignment](https://mlsanity.com/paper/2502.17514/) (2025)
- [Rethinking Data Selection at Scale: Random Selection is Almost All You Need](https://mlsanity.com/paper/2410.09335/) (2024)
- [SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuning](https://mlsanity.com/paper/2405.00705/) (2024)
- [L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models](https://mlsanity.com/paper/2505.17505/) (2025)
- [Exploring the Mystery of Influential Data for Mathematical Reasoning](https://mlsanity.com/paper/2404.01067/) (2024)
- [Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models](https://mlsanity.com/paper/2406.14115/) (2024)
- [Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study](https://mlsanity.com/paper/2408.14438/) (2024)
- [From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuning](https://mlsanity.com/paper/2501.11877/) (2025)
- [OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis](https://mlsanity.com/paper/2501.04561/) (2025)
- [Enabling Weak LLMs to Judge Response Reliability via Meta Ranking](https://mlsanity.com/paper/2402.12146/) (2024)
- [CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization](https://mlsanity.com/paper/2510.17847/) (2025)
- [Curriculum Learning with Quality-Driven Data Selection](https://mlsanity.com/paper/2407.00102/) (2024)
- [HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimation](https://mlsanity.com/paper/2410.05090/) (2024)
- [Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoning](https://mlsanity.com/paper/2505.17266/) (2025)
- [Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning](https://mlsanity.com/paper/2508.21589/) (2025)
- [Baichuan Alignment Technical Report](https://mlsanity.com/paper/2410.14940/) (2024)
- [Distributional Machine Unlearning via Selective Data Removal](https://mlsanity.com/paper/2507.15112/) (2025)
- [Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solving](https://mlsanity.com/paper/2502.12022/) (2025)
- [PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recovery](https://mlsanity.com/paper/2502.12594/) (2025)
- [Rethinking Personalization in Large Language Models at the Token Level](https://mlsanity.com/paper/2603.06595/) (2026)

## Abstract (arXiv)

Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality, inadvertently introducing noise that may compromise model performance. To address this challenge, we introduce textscNuggets, a novel and efficient methodology that leverages one-shot learning to discern and select high-quality instruction data from extensive datasets. textscNuggets assesses the potential of individual instruction examples to act as effective one-shot learning instances, thereby identifying those that can significantly improve performance across diverse tasks. textscNuggets utilizes a scoring system based on the impact of candidate examples on the perplexity of a diverse anchor set, facilitating the selection of the most advantageous data for instruction tuning. Through comprehensive evaluations on two benchmarks, including MT-Bench and Alpaca-Eval, we show that instruction tuning with the top 1% of examples curated by textscNuggets substantially outperforms conventional methods employing the entire dataset.