mlsanityData to 7 Oct 2026

One-Shot Learning as Instruction Data Prospector for Large Language Models

arXiv 2312.10302 ↗First posted 16 Dec 2023cs.CL cs.AIcited by 43

Not on the map. Why?

Abstract (arXiv)

Contemporary practices in instruction tuning often hinge on enlarging data scaling without a clear strategy for ensuring data quality, inadvertently introducing noise that may compromise model performance. To address this challenge, we introduce textscNuggets, a novel and efficient methodology that leverages one-shot learning to discern and select high-quality instruction data from extensive datasets. textscNuggets assesses the potential of individual instruction examples to act as effective one-shot learning instances, thereby identifying those that can significantly improve performance across diverse tasks. textscNuggets utilizes a scoring system based on the impact of candidate examples on the perplexity of a diverse anchor set, facilitating the selection of the most advantageous data for instruction tuning. Through comprehensive evaluations on two benchmarks, including MT-Bench and Alpaca-Eval, we show that instruction tuning with the top 1% of examples curated by textscNuggets substantially outperforms conventional methods employing the entire dataset.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 48 of its 67 references that are in the map. First posted on arXiv 16 Dec 2023.

Builds on

48 of 67 references.

References · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
36 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2023PaLM 2 Technical Reportcited by 1,154
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  11. 2023LIMA: Less Is More for Alignmentcited by 1,080
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  13. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  14. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  15. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  16. 2023Instruction Tuning with GPT-4cited by 676
  17. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  18. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  19. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  20. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  21. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  22. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  23. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  24. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  25. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  26. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  27. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  28. 2023On the Exploitability of Instruction Tuningcited by 110
  29. 2023Finding Support Examples for In-Context Learningcited by 104
  30. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
  31. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  32. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  33. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignmentcited by 33
  34. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  35. 2022MVP: Multi-task Supervised Pre-training for Natural Language Generationcited by 16
  36. 2023Iterative Forward Tuning Boosts In-Context Learning in Language Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 14; 2026: 6.

43 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  3. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  4. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 92
  5. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 76
  6. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  7. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selectioncited by 48
  8. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
  9. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  10. 2024SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflectioncited by 34
  11. 2024Rethinking Data Selection at Scale: Random Selection is Almost All You Needcited by 22
  12. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 22
31 more
  1. 2024SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuningcited by 19
  2. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  3. 2024Exploring the Mystery of Influential Data for Mathematical Reasoningcited by 13
  4. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Modelscited by 12
  5. 2024Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Studycited by 12
  6. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  7. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  8. 2024Enabling Weak LLMs to Judge Response Reliability via Meta Rankingcited by 9
  9. 2025CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimizationcited by 8
  10. 2024Curriculum Learning with Quality-Driven Data Selectioncited by 7
  11. 2024HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimationcited by 7
  12. 2025Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoningcited by 7
  13. 2025Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learningcited by 7
  14. 2024Baichuan Alignment Technical Reportcited by 6
  15. 2025Distributional Machine Unlearning via Selective Data Removalcited by 5
  16. 2025Teaching LLMs According to Their Aptitude: Adaptive Reasoning for Mathematical Problem Solvingcited by 4
  17. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 3
  18. 2026Rethinking Personalization in Large Language Models at the Token Levelcited by 3
  19. 2026Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methodscited by 3
  20. 2026MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selectioncited by 3
  21. 2024Small Language Model as Data Prospector for Large Language Modelcited by 2
  22. 2025ZiGong 1.0: A Large Language Model for Financial Creditcited by 2
  23. 2025LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selectioncited by 2
  24. 2024Data Quality Control in Federated Instruction-tuning of Large Language Modelscited by 1
  25. 2024HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarioscited by 1
  26. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  27. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMscited by 1
  28. 2026On Representation Redundancy in Large-Scale Instruction Tuning Data Selectioncited by 1
  29. 2024Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learningcited by 0
  30. 2024Learning from "Silly" Questions Improves Large Language Models, But Only Slightlycited by 0
  31. 2026Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxiescited by 0