mlsanityData to 7 Oct 2026

Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning

arXiv 2305.09246 ↗First posted 16 May 2023cs.AI cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

Instruction tuning for large language models (LLMs) has gained attention from researchers due to its ability to unlock the potential of LLMs in following instructions. While instruction tuning offers advantages for facilitating the adaptation of large language models (LLMs) to downstream tasks as a fine-tuning approach, training models with tens of millions or even billions of parameters on large amounts of data results in unaffordable computational costs. To address this, we focus on reducing the data used in LLM instruction tuning to decrease training costs and improve data efficiency, dubbed as Low Training Data Instruction Tuning (LTD Instruction Tuning). Specifically, this paper conducts a preliminary exploration into reducing the data used in LLM training and identifies several observations regarding task specialization for LLM training, such as the optimization of performance for a specific task, the number of instruction types required for instruction tuning, and the amount of data required for task-specific models. The results suggest that task-specific models can be trained using less than 0.5% of the original dataset, with a 2% improvement in performance over those trained on full task-related data.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 26 of its 37 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

26 of 37 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
14 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  8. 2022Galactica: A Large Language Model for Sciencecited by 736
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  10. 2022Prompting GPT-3 To Be Reliablecited by 258
  11. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  12. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  13. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  14. 2022How Many Data Samples is an Additional Instruction Worth?cited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 21; 2025: 16; 2026: 6.

56 papers.

Sort:

By area:

  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 76
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  5. 2023TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPTcited by 57
  6. 2023From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Modelscited by 45
  7. 2024Learning or Self-aligning? Rethinking Instruction Fine-tuningcited by 43
  8. 2023WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuningcited by 42
  9. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 33
  10. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  11. 2024TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Datacited by 28
  12. 2025AnomalyR1: A GRPO-based End-to-end MLLM for Industrial Anomaly Detectioncited by 28
38 more
  1. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  2. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  3. 2023LARP: Language-Agent Role Play for Open-World Gamescited by 24
  4. 2025Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metriccited by 24
  5. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  6. 2024RECOST: External Knowledge Guided Data-efficient Instruction Tuningcited by 18
  7. 2024Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Modelscited by 18
  8. 2023Data Management For Training Large Language Models: A Surveycited by 17
  9. 2025A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigmscited by 17
  10. 2024SelectLLM: Can LLMs Select Important Instructions to Annotate?cited by 16
  11. 2024SMART: Submodular Data Mixture Strategy for Instruction Tuningcited by 16
  12. 2023Becoming self-instruct: introducing early stopping criteria for minimal instruct tuningcited by 13
  13. 2025Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curationcited by 13
  14. 2025Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencodercited by 11
  15. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 10
  16. 2024Federated Data-Efficient Instruction Tuning for Large Language Modelscited by 9
  17. 2025A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solvingcited by 9
  18. 2023Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Modelscited by 6
  19. 2023DavIR: Data Selection via Implicit Reward for Large Language Modelscited by 6
  20. 2024IterSelectTune: An Iterative Training Framework for Efficient Instruction-Tuning Data Selectioncited by 6
  21. 2024Baichuan Alignment Technical Reportcited by 6
  22. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Modelscited by 5
  23. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  24. 2025Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuningcited by 4
  25. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  26. 2024A Bayesian Approach to Data Point Selectioncited by 3
  27. 2026Training-Trajectory-Aware Token Selectioncited by 3
  28. 2026A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)cited by 3
  29. 2023Constituency Parsing using LLMscited by 2
  30. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Modelscited by 2
  31. 2024Align2LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curationcited by 2
  32. 2024From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmapcited by 2
  33. 2025ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Modelscited by 2
  34. 2025LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selectioncited by 2
  35. 2024Data Quality Control in Federated Instruction-tuning of Large Language Modelscited by 1
  36. 2025Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillationcited by 1
  37. 2026Measuring Distribution Shift in User Prompts and Its Effects on LLM Performancecited by 1
  38. 2026Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Driftcited by 1

50 most cited of 56 All 56 (JSON)